Cannot pull images when logged in to Docker Desktop (Status 500: unauthorized: incorrect username or password)

[MH-17]

Test Containers 1.16.3 on Docker Engine 20.10.12 with Docker Desktop using WSL2 Backend on Windows 21H1.

When I am logged in to my account in Docker Desktop, Test Containers can no longer pull images from Docker Hub. Once I log out of my account, Test Containers is able to pull images as expected.

com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.3\": unauthorized: incorrect username or password"}
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

I think my Docker installation is fine, as I can pull images from the terminal when I am logged in to Docker Desktop. I don't want to log out every time I use Test Containers. How can I fix this issue?

Logs (on INFO): https://pastebin.com/fyExxi76

Logs (on DEBUG): https://pastebin.com/b4H283fL

[luedi]

I have exactly the same issue with Test Container 1.16.2. This issue only occurs if the Docker image has to be pulled. As long as I have a local image everything is fine.

[awojcik64]

Problem still exists with Test Containers 1.16.3 and Docker Desktop with versions: 4.2.0, 4.3.2, 4.6.1 and 4.7.0
Actually, it happens regardless if I'm logged in or not.

12:21:49.321 [main] INFO  org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
12:21:49.368 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.NpipeSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
12:21:49.854 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Npipe socket (npipe:////./pipe/docker_engine)
12:21:49.854 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is localhost
12:21:49.888 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.14
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 51188 MB
12:21:55.569 [docker-java-stream--301690517] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.3\": unauthorized: incorrect username or password"}

Image pulled manually via docker pull testcontainers/ryuk:0.3.3 works just fine. It's just a workaround though.

[kiview]

Thanks for bumping this issue. I was recently running into this error on Docker Desktop as well.
Can you please try out to logout and then perform the Docker login on the command line instead (docker login)?

And can someone reproduce this with Docker Desktop on macOS as well?

[SiwyDym]

I tested on 4.3.2 on macOS(12.1 (21C52)) and it's looking like a charm, but for Windows I encountered the same issue as @awojcik64

[awojcik64]

I don't know how I did that, but now containers are pulled correctly even if I am logged out of everything.
I was only juggling logging in using GUI and command line. (tested on Docker Desktop 4.7.0)

[kiview]

@awojcik64 How I perceive the error so far with regards to pulling public images with Testcontainers:

• Docker Desktop did the login through the GUI: not working ❌
• Docker login through CLI and username/pw (docker login): working ✅
• Docker logged out: working ✅

For further triangulation, may I ask you @SiwyDym if you are sure that on macOS, you recently did a login through the Docker Desktop GUI and it still worked? Also, please make sure the images are not cache before trying out Testcontainers.

So as a current workaround, I'd suggest doing the docker login from CLI instead of the GUI and everything should work fine.

[SiwyDym]

I can confirm that issue occurs when I login by Docker Desktop GUI :|
Docker Desktop: 4.7.0 (77141)
OS: 12.1 (21C52)

[kiview]

@SiwyDym Can you confirm that performing the login through CLI mitigates the issue?

[SiwyDym]

I cannot confirm that, the only working mitigation for me is manually pulling the image testcontainers/ryuk:0.3.3

[kiview]

@SiwyDym I see. Can you instead confirm that it works if your Docker daemon is unautheticated? (after docker logout)

[kiview]

Another interesting aspect to try out is if doing a docker login registry.hub.docker.com mitigates the situation.

[jodastephen]

I'm running Docker Desktop 4.7.0 (77141) on Windows 21H2. As a new user, I was completely unable to find any workaround from the selection above. I tried logging out, logging in via the UI, logging in via the command line, logging into registry.hub.docker.com, nothing worked.

I then went to Docker Desktop and clicked "Reset to factory defaults" and it worked perfectly first time (not logged in). I tried deleting the images and it re-download them perfectly.

However, when I then logged in using the command line and removed the images, it failed again. Once I reset Docker Desktop again, everything worked again.

Hope that helps.

[kiview]

Thanks a lot for sharing your experiences @jodastephen, this really helps in triaging this issue. I am currently in the process of debugging and triaging as well.

What seems to help is to, in addition to performing a docker logout, also remove the Docker config file:

rm USER_HOME\.docker\config.json

I currently suspect this is caused by changes to the Docker Hub registry, but the permutation in which this is currently failing are unfortunately not 100% clear to me yet.

[wetted]

I am also having this exact issue. I am setting up a new development environment on a M1 Max MacBook Pro, with latest Docker (darwin/arm64). I have tried all the suggestions in the previous comments; i.e. logging out of docker, logging in via CLI (with and without registry.hub.docker.com), reset Docker to factory defaults, delete .docker/config.json.

The only thing that ever works for me is to manually pull the image testcontainers/ryuk:0.3.3

Docker Desktop 4.7.0 (77141)
Docker version 20.10.14, build a224086
macOS 12.3.1 (21E258)

[kiview]

Would someone affected by this issue like to try out this PR from Jitpack?
#5263

https://www.testcontainers.org/jitpack_dependencies/

[wetted]

Would someone affected by this issue like to try out this PR from Jitpack? #5263

https://www.testcontainers.org/jitpack_dependencies/

This worked for me running the test from the Micronaut Data-JDBC guide. I used the following dependencies from jitpack:

    testImplementation "com.github.testcontainers.testcontainers-java:testcontainers:9baedef"
    testImplementation "com.github.testcontainers.testcontainers-java:mysql:9baedef"
    testImplementation "com.github.testcontainers.testcontainers-java:junit-jupiter:9baedef"

[jodastephen]

This appears to work for me, thanks.

[kiview]

Based on the idea behind the PR I tried the following workaround with the current release and it works for me on Windows:

docker login index.docker.io

Would anyone affected like to give it a try as well?

[wetted]

That worked for me on M1 Macbook Pro. It added an {"auths":{"index.docker.io":{}}} entry in my ~/.docker/config.json file. Thereafter the workaround persists across, restarts, etc.

[dermoritz]

i tried resetting docker windows setup to defaults, login / logout with docker (and specific registry urls) nothing helped.

The only thing that helped me was pulling every image needed manually.

But there was something odd: The first image needed was ryuk (one project 0.3.3 another 0.3.1) but after pulling only this and run the project again i get too many failed login requets for the other images to pull (for ryuk i always got unauthorized).

[kiview]

@dermoritz So you say, login with exactly docker login index.docker.io did not help?

[mira-silhavy]

@kiview I can also confirm the command fixes this for me.

[teddypickerfromul]

Can confirm the same issue with M1 Mac, Docker Desktop 4.8.1 (78998), macOS 12.3.1, testcontainers 1.17.1

Nothing mentioned up here doesn't help - still getting:

Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://registry-1.docker.io/v2/library/postgres/manifests/14.2-alpine\": toomanyrequests: too many failed login attempts for username or IP address"}

Tried to pull test image manually - no way :(

[kiview]

You are now getting this error from registry:

toomanyrequests: too many failed login attempts for username or IP address"

That's why the workaround did not work. You have to wait until the block times out on Docker Hub.

[teddypickerfromul]

You are now getting this error from registry:

toomanyrequests: too many failed login attempts for username or IP address"

That's why the workaround did not work. You have to wait until the block times out on Docker Hub.

Still waiting for several (5?) days, so...

[kiview]

Which is the workaround you have currently applied? Also, maybe the block does not time out if you are connecting from the same IP address.

[jim-jobcase]

@kiview I just tried the command line fix on macOS 12.3.1 (M1 Pro) running Docker Desktop 4.8.1 and confirmed it works here too.

Thank you!

Edit: This was also using Testcontainers 1.16.2

[grishick]

The fix that worked for me on an M1 Mac was:
remove credsStore:desktop from ~/.docker/config.json, run docker login in terminal

[StFS]

I'm running into this problem in our CI environment.

We're running testcontainers via maven and this is the command line we use to run maven:
mvn verify -DskipUnitTests -Dregistry.url=${CI_REGISTRY} -Dregistry.username=${CI_REGISTRY_USER} -Dregistry.password=${CI_REGISTRY_PASSWORD}

We need to use a private registry to get the containers that are being tested, but for some reason, this seems to break testcontainers ability to fetch public containers:

119153 [main] WARN  🐳 [testcontainers/ryuk:0.3.4]  - Retrying pull for image: testcontainers/ryuk:0.3.4 (2s remaining)
124727 [docker-java-stream--909396508] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate  - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Get https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.4: unauthorized: incorrect username or password"}
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

Since the container that we're running the Maven tests from within doesn't have the docker executable, I can't run docker login in our CI scripts.

Does anybody have anything I could try out to get this working?

[kiview]

@StFS Can you please ask about this in Slack, or open a dedicated discussion/issue?
This issue is about Docker Desktop.

[StFS]

@StFS Can you please ask about this in Slack, or open a dedicated discussion/issue? This issue is about Docker Desktop.

Good point. Sorry about that.

[OlehOnyshchuk]

Had the same issue. Logout, and than login via console. Helped for me.

1. docker logout
2. docker login --username

[cdlarue]

I'm running into this problem in our CI environment.

We're running testcontainers via maven and this is the command line we use to run maven: mvn verify -DskipUnitTests -Dregistry.url=${CI_REGISTRY} -Dregistry.username=${CI_REGISTRY_USER} -Dregistry.password=${CI_REGISTRY_PASSWORD}

We need to use a private registry to get the containers that are being tested, but for some reason, this seems to break testcontainers ability to fetch public containers:

119153 [main] WARN  🐳 [testcontainers/ryuk:0.3.4]  - Retrying pull for image: testcontainers/ryuk:0.3.4 (2s remaining)
124727 [docker-java-stream--909396508] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate  - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Get https://registry-1.docker.io/v2/testcontainers/ryuk/manifests/0.3.4: unauthorized: incorrect username or password"}
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:833)

Since the container that we're running the Maven tests from within doesn't have the docker executable, I can't run docker login in our CI scripts.

Does anybody have anything I could try out to get this working?

This is the exact situation I have as well. Has anyone found a workaround or solution?

[angelomelonas]

I am also encountering this issue while running Github Actions locally using Act. I have tried all the suggestions above, and none seem to work. When running my tests in CI (Github Actions), it works fine though.

[josebarros2025]

I have exactly the same issue with Test Container 1.16.2. This issue only occurs if the Docker image has to be pulled. As long as I have a local image everything is fine.

sorry this isnt a valid solution, somebody must run this in a pipeline for example, the image must be pulled

[kiview]

@angelomelonas We never tested TC with Act, but essentially, this will lead to Docker-in-Docker scenario, so the issue you are facing here, is very likely independent from the issue that is discussed here. I would suggest opening a dedicated issue or discussion for using TC with Act and Docker authentication.

[angelomelonas]

Ah that does makes sense. I will create a separate issue for that. Thanks @kiview!