Devcontainer version 0.275 - auto adding ~/.docker/config.json with credsStore specified which cause az acr login to fail

[indra-cloudwall]

Does this issue occur when all extensions are disabled?: Yes/No

• VS Code Version:
• OS Version:

Steps to Reproduce:

1. Start dev container using version 0.275
2. run az acr login, it will fail with below error message:
   

[KUGA2]

Workaround: clean credsStore from ~/.docker/config.json and rollback the DevContainer extension.

[indra-cloudwall]

@KUGA2 yep, that what I did, but hopefully this will get fixed

[chrmarti]

Are you running az acr login inside the container? Does az acr login work locally?

Do you mount the local ~ or ~/.docker in the container?

Could you append the log from when this happens? (F1 > Dev Containers: Show Container Log)

[thomet]

Hey, I'm having basically the same issue, but with the ecr-login helper. I have used the ecr helper inside the container, but as the ~/.docker/config.json got now overwritten, I always need to manually fix that. I would be ok using forwarding to my local, but also this isn't working well. I have configured the ecr-login for my remote in the config.json on my local machine like this:

{
  "auths": {},
  "credHelpers": {
    "??????????????.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login"
  },
  "credsStore": "desktop"
}

But the forward to the devcontainer isn't working. Only if I set credsStore to ecr-login. But Docker Desktop is overwriting the credsStore and set it do desktop on each update. Would be good to have the following improvements:

• support credHelpers setting on forward
• add option to disable to forwarding in case you do not want it

[aservedio]

I think the issue I have is the same one reported here.

I have a project where the docker login happens inside the devcontainer after it's built and after the user logged in to the secret managed tool where the docker credentials are located.

Since the recent update, I am unable to docker login to my private registry or docker pull from it.

Docker credential helper not found: Exectuable 'docker-credential-wincred' not found on PATH 'C:\ProgramData\etc-...'..Login Succeeded
Docker credential helper not found: Exectuable 'docker-credential-wincred' not found on PATH 'C:\ProgramData\etc-...'..Error response from daemon: Head "https://my-own-private-registry": denied: access forbidden

I noticed an added "credsStore": "dev-containers-daed49da10eefe6b94db768833de9eaaf5f681d3" added inside devcontainer ~/.docker/config.json, and if I delete it all my problems goes away. Any option to disable it alltogether kind of like what we can already do with git credential helper?

~/.docker or the config.json files are not mounted, and I don't use host level credentials in this project (which has it's own secrets manager where the docker credentials are pulled from).

This is what shows up in the container log when the error happens:

[47499 ms] Host server: Error: spawn docker-credential-secretservice ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
[47501 ms] Host server: (node:20970) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)
(Use `node --trace-warnings ...` to show where the warning was created)
[47702 ms] Host server: Error: spawn docker-credential-secretservice ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:478:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
[47703 ms] Host server: (node:20970) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 2)

Meanwhile: cat ~/.docker/config.json | jq 'del(.credsStore)'

[indra-cloudwall]

Are you running az acr login inside the container? Does az acr login work locally?

Do you mount the local ~ or ~/.docker in the container?

Could you append the log from when this happens? (F1 > Dev Containers: Show Container Log)

• Yes, running az acr login inside the container does not work while running it locally works

• It is ~/.docker in the container

• There is no log generated from (F1 > Dev Containers: Show Container Log). The only error is from the terminal running the az acr login (the original screenshot I provided)

[chrmarti]

@thomet Thanks, I will fix the existing lookup for credHelpers. FYI: Not sure if this is needed, but I see https://github.com/awslabs/amazon-ecr-credential-helper#configuration suggests an additional entry for public.ecr.aws.

@aservedio I will add a user setting. Additionally I can check if there is already a credStore set and leave that unchanged (though I guess you don't have one set if I understand correctly).

@indra-cloudwall Could you post your ~/.docker/config.json with any private information replaced with placeholders (to preserve the file structure)?

[aservedio]

@chrmarti So on my WSL host I had no ~/.docker/config.json, and in the devcontainer the ~/.docker/config.json file contains only the credStore entry, until the docker login command which seems to succeed despire the credStore helper error messages, but then the pull fails.

Another thing I just noticed few minutes ago on a different devcontainer that has no docker login to any private registry, I got this error msg from credStore while pulling a public image after the devcontainer started:

docker pull linuxkit/containerd:eeb3aaf497c0b3f6c67f3a245d61ea5a568ca718-amd64
Docker credential helper not found: Exectuable 'docker-credential-wincred' not found on PATH 'C:\bunch-of-sture...'..eeb3aaf497c0b3f6c67f3a245d61ea5a568ca718-amd64: Pulling from linuxkit/containerd
Digest: sha256:c8175f26968f8641fc47f02979771aafee4c8ebe505c8ab65e1f0967a7d66adc
Status: Image is up to date for linuxkit/containerd:eeb3aaf497c0b3f6c67f3a245d61ea5a568ca718-amd64
docker.io/linuxkit/containerd:eeb3aaf497c0b3f6c67f3a245d61ea5a568ca718-amd64

So basically just noise, but makes me wonder if this helper if taking a more opiniated approach compared to git credential helper.

[aservedio]

Forgot to answer the question: i think I don't mind the credential helper but it seems to get in the way with credentials already handled in that config.

[chrmarti]

Released 0.278.0-pre-release with the following changes:

• Fix credHelpers lookup. (Should address @thomet's case.)
• Handle missing binary like missing credentials. (Should avoid the noise in the log @aservedio mentions.)
• Do not overwrite existing credStore entry.
• Add user setting to turn off the Docker credentials helper completely. (Please try with the above improvements first.)

Could everyone give this version a try and let me know if this fixes the Docker credentials helper for you? Thanks!

[aservedio]

One last new thing I noticed related to this issue, on a 3rd different project. I can't run docker build within the devcontainer until I remove credsStore entry.

⬢ [Docker] ❯ docker build -t dev-containerd .
[+] Building 0.3s (2/3)
 => [internal] load build definition from Dockerfile 0.1s
[+] Building 0.4s (3/3) FINISHED
 => [internal] load build definition from Dockerfile 0.1s
 => => transferring dockerfile: 6.30kB 0.0s
 => [internal] load .dockerignore 0.1s
 => => transferring context: 2B 0.0s
3 => ERROR [internal] load metadata for docker.io/library/golang:1.19.5 0.3s
e------C:\Users\x\AppData\Local\Microsoft\WindowsApps;C:\Users\x\AppData\Local\etc.....'..
> [internal] load metadata for docker.io/library/golang:1.19.5:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: rpc error: code = Unknown desc = EOF

In that devcontainer, the content of ~/.docker/config.json is:

{
        "credsStore": "dev-containers-somehash"
}

[aservedio]

Here's my test results on the 3 bugs I added above, with v0.278.0 pre-release:

unable to docker build

Got an error but cleaner I think.

[+] Building 0.4s (3/3) FINISHED
 => [internal] load build definition from Dockerfile 0.0s
 => => transferring dockerfile: 6.30kB 0.0s
 => [internal] load .dockerignore 0.1s
 => => transferring context: 2B 0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:1.19.5 0.3s
------
 > [internal] load metadata for docker.io/library/golang:1.19.5:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: rpc error:
  code = Unknown desc = error getting credentials - err: exit status 1,
  out: `Docker credential helper not found: Exectuable 'docker-credential-wincred' not found on PATH 'blabla'..`

works if I add "dev.containers.dockerCredentialHelper": false to devcontainer.json

unable to docker pull public image

That one I tried without disabling credential helper, confirmed it got added in config.json, and public pulls works fine this time! 👍

unable to docker login & docker pull on private registry

This one still fails but I got slightly different behavior. Docker login to my private registry failed more explicitely and failed the docker login. Initially the docker login succeeded while the error was still outputted, and with the valid docker login in the config the pull then failed. Now it stops at docker login which is better then before for sure.

Error saving credentials: error storing credentials - err: exit status 1, out: `Docker credential helper not found: Exectuable 'docker-credential-wincred' not found on PATH 'C:\blablable`

So maybe this is a 4th kind of bug I'm reporting? Credential helper crash usage of docker login in a devcontainer.

[johnDeSilencio]

Released 0.278.0-pre-release with the following changes:

• Fix credHelpers lookup. (Should address @thomet's case.)
• Handle missing binary like missing credentials. (Should avoid the noise in the log @aservedio mentions.)
• Do not overwrite existing credStore entry.
• Add user setting to turn off the Docker credentials helper completely. (Please try with the above improvements first.)

Could everyone give this version a try and let me know if this fixes the Docker credentials helper for you? Thanks!

Hi there, @chrmarti! I'm experiencing this bug as well. I'd like to help and test out the 0.278.0-pre-release as you've requested. I'm a long-time VSCode user, but I am very green to the extension ecosystem. Where can I find the VSIX file for the 0.278.0-pre-release of the Dev Containers extension? Or is there some other way I'm supposed to install it?

EDIT: I found the "Switch to Prerelease Version" option for the Dev Containers extension! I can confirm that that 0.278.0-pre-release fixes the issue for me. VSCode no longer overwrites credsStore in ~/.docker/config.json and instead uses the Docker credential manager that has already been configured (I use pass).

[thomet]

@chrmarti I have switched to the prebuild and can verify:

• Verified existing ~/.docker/config.json will not overwrite anymore in devcontainer
• Verified that credHelpers config will be used when using docker credentials forwarding to local system

My problem would be solved with this version :) Thanks

[chrmarti]

@aservedio Dev Containers 0.279.0-pre-release comes with fixes for the docker build and docker login issues you saw with 0.278.0-pre-release. Would be great if you could give that a try.

The main fix is that we first check if the configuration on the local machine has any credential helpers or auth tokens configured and if not, we don't register the credential helper in the container.

If anyone else still sees issues with 0.279.0-pre-release, please let me know.

[chrmarti]

Thanks everyone for the great feedback. Closing as fixed, let me know if there are any remaining issues with the latest pre-release version.

[johnstairs]

The latest prerelease version doesn't work for me either. In my setup, I'm using devcontainers over SSH. az acr login now gives me

Error saving credentials: error storing credentials - err: exit status 1, out: `Docker credential helper 'docker-credential-osxkeychain' not found: write EPIPE.`

It looks like it is looking for the credential helper on my local mac, not the Linux host I'm SSHing into, where the devcontainer is actually running, which has no credential helper.

[KUGA2]

Even in preview version (v.0.281.0) it still keeps adding

"credsStore": "dev-containers-c43e2fdf-7a18-4fc5-bd0b-61ea8b1e0e61"

to my ~/.docker/config.json file on the host. Next switch to container then fails.

Working on wsl. Let me know if you need any more logs to investigate this.

[finite8]

This is still an issue for me. Everytime the extension starts up it adds "credsStore" in and breaks docker login functionality. I tried this with the current version as well as the 0.278.0-pre-release.

Is it possible to turn this feature off in the devcontainer.json?

[finite8]

Thanks everyone for the great feedback. Closing as fixed, let me know if there are any remaining issues with the latest pre-release version.

@chrmarti
This is still an issue. Could we have this ticket reopened or a new one raised please?

[chrmarti]

Tracking the option to turn the entire feature off in #8201. Thanks.

[chrmarti]

The setting appears to work for me. Could everyone still seeing problems with Dev Containers 0.286.0-pre-release open new issues?