Mac OS X: (docker? git?) credentials and keychain failure

[richarddli]

I run forge deploy of the todo app against an empty cluster. Not all of my services deploy. I repeat forge deploy. More services deploy. I run it again. Finally, all my services deploy.

The error messages I get from the first run are below (I get them in the second run as well, but not in the final run).

MacBook-Pro-23:todo richard$ forge deploy
forge: 3 child task(s) errored
  scan: /Users/richard/dw/todo -> api, auth, todo-db, prometheus, search, tasks
  dependencies: api, auth, todo-db, prometheus, search, tasks ->
  service: api -> 1 child task(s) errored
    build: api:052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: api 052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: auth -> 1 child task(s) errored
    build: auth:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: auth 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: todo-db
    build: manifests /Users/richard/dw/todo/work/k8s/todo-db
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/todo-db -> OK
      storageclass "fast" created
      service "todo-db" created
      statefulset "todo-db" created
  service: prometheus
    build: pushed registry.hub.docker.com/richarddli/prometheus:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git
      manifests /Users/richard/dw/todo/work/k8s/prometheus
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/prometheus -> OK
      service "prometheus" created
      deployment "prometheus" created
      configmap "prometheus-config" created
  service: search
    build: manifests /Users/richard/dw/todo/work/k8s/search
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/search -> OK
      service "search" created
      deployment "search" created
  service: tasks -> 1 child task(s) errored
    build: tasks:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: tasks 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`

[ark3]

This looks like a Mac-specific issue with git and the macOS keychain credential helper, which Apple's Git uses by default, I believe. I can help you debug this. It would be useful to know what subcommands are being run that generate those errors.

><> fgrep -B1 keychain ~/.gitconfig
[credential]
  helper = osxkeychain
><> git credential-osxkeychain
usage: git credential-osxkeychain <get|store|erase>

[rhs]

There should be a log of all the commands in /tmp/forge.log

[richarddli]

`Step 1/4 : FROM datawire/ambassador-envoy:latest
---> f8e53d525e68
Step 2/4 : RUN apt-get update && apt-get -q install -y curl dnsutils
---> Using cache
---> f77fc9f9afaf
Step 3/4 : ENTRYPOINT /usr/local/bin/envoy
---> Using cache
---> 9adb9a03eef9
Step 4/4 : CMD -c /etc/envoy/envoy.json
---> Using cache
---> b07d55aae0ed
Successfully built b07d55aae0ed
Successfully tagged registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git
(0:00:01)
INFO forge[1].service[1].build[1].bake[1].docker-build[1]: RESULT -> None (0:00:01)
INFO forge[1].service[1].build[1].bake[1]: RESULT -> ['Dockerfile'] (0:00:03)
INFO forge[1].service[1].build[1].push[1]: START(api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1]: checking if api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git containers exist
INFO forge[1].service[1].build[1].push[1].applicator[1]: START((api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git, 'api', 'Dockerfile'))
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: START(docker, images, -q, registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: docker images -q registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc3... -> (in progress)
b07d55aae0ed

INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: docker images -q registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc3... -> b07d55aae0ed

INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: RESULT -> b07d55aae0ed
(0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].remote_exists[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].remote_exists[1]: RESULT -> False (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1]: pushing container Dockerfile
INFO forge[1].service[1].build[1].push[1].push[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: START(docker, login, -u, richarddli, -p, , registry.hub.docker.com)
INFO forge[1].service[4].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[4].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1]: RESULT -> ERROR (0:00:05)
INFO forge[1].service[4]: RESULT -> ERROR (0:00:05)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Login Succeeded

INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> Login Succeeded

INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: RESULT -> Login Succeeded
(0:00:02)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: START(docker, push, registry.hub.docker.com/richarddli/search:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]

INFO forge[1].service[3].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[3].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[3]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[2].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[2].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[2]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Login Succeeded

INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> Login Succeeded

INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: RESULT -> Login Succeeded
(0:00:02)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[2]: START(docker, push, registry.hub.docker.com/richarddli/tasks:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/tasks:8587e3eb8701ef5d... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/tasks]

INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]
9b01d945127f: Preparing

INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]
9b01d945127f: Preparing
77eaac6d0b1f: Preparing
`

[ark3]

@richarddli Does the behavior change if you turn off "Securely store docker logins in macOS keychain" in Docker for Mac Preferences/General?

[tristanpemble]

@ark3 I've seen this error come and go, and after disabling that setting in Docker for Mac it seemed to go away

[richarddli]

I haven't run Forge extensively since changing the setting but it does seem to make it go away.

[rhs]

I found this issue that looks like it might be the same when googling this error: docker/for-mac#1540

It talks about removing "credsStore": "osxkeychain" from ~/.docker/config.json. Does anyone (or can someone with a mac check) if this is the same thing that the "Docker for Mac Preferences/General" workaround is tweaking?

If it is, then I might be able to fix this by launching docker with a custom config setup rather than letting it point to the default ~/.docker/config.json. This would have other UX implications though, e.g. any other config customizations someone might have made in that file wouldn't take effect for forge. I don't know how big a deal this is offhand. (I don't personally customize docker behavior all that much.) Alternatively I could detect the setting and/or the error message when this happens and warn/point to the workaround.

[richarddli]

So if the box is checked, the config.json file says:

{
  "auths" : {
    "registry.hub.docker.com" : {

    },
    "https://registry.hub.docker.com" : {

    }
  },
  "credsStore" : "osxkeychain"

If it is not checked, it reads:

{
	"auths": {
		"https://registry.hub.docker.com": {
			"auth": "cmljaGi89dfhjasdhjfhjdfhjadfhjhsdfdfU"
		},
		"registry.hub.docker.com": {
			"auth": "cmljaGi89dfhjasdhjfhjdfhjadfhjhsdfdfU"
		}
	}

(these are not my actual credentials)

[plombardi89]

I'm really not a fan of the idea of using a different Docker config file than the one used by default. It means I now need to maintain the logins to registries for Forge as well as when I am doing out of band Forge work involving push/pull from authorized or private registries.

[richarddli]

I added documentation about configuring Docker for Mac as a temporary (and permanent?) solution.

[romualdr]

MacOS Version 17.09.0-ce-mac35 (19611)

Docker login failed every time with "Securely store docker logins in macOS keychain"

error getting credentials - err: exit status 1, out: The user name or passphrase you entered is not correct.`

I turned the setting off and it worked as expected.

This setting is hell.