DockerRegistry block throws ImageNotFound: 404 Client Error despite correct authentication to private container registry

[anna-geller]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

It doesn't seem like an issue introduced in the latest release only as I was able to reproduce the error on 2.2 and 2.1 too.

Problem: authentication with private Docker registries doesn't seem to work - reproduced with a private DockerHub registry

Reproduction

Block creation: https://github.com/anna-geller/prefect-docker-deployment/blob/main/blocks/dockerhub_block.py

Deployment:

python blocks/dockerhub_block.py
prefect deployment build flows/healthcheck.py:healthcheck -n prod -q prod -ib docker-container/prod --apply
prefect deployment run healthcheck/prod

Error

No response

Versions

Version:             2.3.0
API version:         0.8.0
Python version:      3.9.12
Git commit:          8d9316c0
Built:               Tue, Aug 30, 2022 5:30 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         hosted

Additional context

might be fixed in #6622

[sti0]

I introduced this issue on SLACK so I just want to add my agent log which prints out a a internal server error ("no basic auth credentials").

Starting v2.3.0 agent with ephemeral API...

  ___ ___ ___ ___ ___ ___ _____     _   ___ ___ _  _ _____
 | _ \ _ \ __| __| __/ __|_   _|   /_\ / __| __| \| |_   _|
 |  _/   / _|| _|| _| (__  | |    / _ \ (_ | _|| .` | | |
 |_| |_|_\___|_| |___\___| |_|   /_/ \_\___|___|_|\_| |_|


Agent started! Looking for work from queue(s): default...
15:24:30.641 | INFO    | prefect.agent - Submitting flow run '6acdbe6c-56f7-49fa-bf61-792d0f1d3df9'
15:24:31.795 | INFO    | prefect.infrastructure.docker-container - Pulling image '<<url>>/<<image>>:prefect2'...
15:24:32.990 | ERROR   | prefect.agent - Failed to submit flow run '6acdbe6c-56f7-49fa-bf61-792d0f1d3df9' to infrastructure.
Traceback (most recent call last):
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/images/create?tag=prefect2&fromImage=<<url>>%2F<<image>>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/agent.py", line 206, in submit_run
    await self.task_group.start(submit_flow_run, flow_run, infrastructure)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in start
    return await future
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/infrastructure/submission.py", line 48, in submit_flow_run
    return await infrastructure.run(task_status=task_status)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 239, in run
    container_id = await run_sync_in_worker_thread(self._create_and_start_container)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 281, in _create_and_start_container
    self._pull_image(docker_client)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/prefect/infrastructure/docker.py", line 392, in _pull_image
    return docker_client.images.pull(image, tag)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/docker/models/images.py", line 465, in pull
    pull_log = self.client.api.pull(
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/docker/api/image.py", line 429, in pull
    self._raise_for_status(response)
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e) from e
  File "/workspace/tmp/venv/prefect2/lib/python3.10/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=prefect2&fromImage=<<url>>%2F<<image>>: Internal Server Error ("Head "https://<<url>>/v2/<<image>>/manifests/prefect2": no basic auth credentials")

Using the same credentials with docker login... works totally fine. We are hosting our registry on-prem with Nexus 3, which should make no difference dockerhub example from @anna-geller, because we are specifing the url inside the Docker Registry block.
When inspecting the Docker Container block the registry is visible:

Version

Version:             2.3.0
API version:         0.8.0
Python version:      3.10.4
Git commit:          8d9316c0
Built:               Tue, Aug 30, 2022 5:30 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.34.1

[jegesh]

I'm also experiencing this. In fact, I get the 404 even if the docker image is already available locally. Very frustrating

[tonal]

How to about Prefect 2.4.x?

[nibrag]

How to about Prefect 2.4.x?

The same.

It happens because DockerContainer and DockerRegistry have different docker_client.
And when DockerContainer pulls image it calls DockerContainer.image_registry.login(), so authentication passes successfully inside DockerContainer.image_registry but then DockerContainer still uses its own docker-client which is still without authentication.

[sti0]

@madkinsz is there any timeline when this issue will be fixed? We're waiting currently for a fix to complete our migration to Prefect v2.

[anna-geller]

@sti0 can you check of this PR fixes it for you? #6889

[sti0]

@anna-geller I can confirm that #6889 fixes the problem for me. I can run flows as docker container infrastructure from a private registry 🚀