New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mac OS X: (docker? git?) credentials and keychain failure #28

Closed
richarddli opened this Issue Aug 17, 2017 · 11 comments

Comments

Projects
None yet
6 participants
@richarddli
Contributor

richarddli commented Aug 17, 2017

I run forge deploy of the todo app against an empty cluster. Not all of my services deploy. I repeat forge deploy. More services deploy. I run it again. Finally, all my services deploy.

The error messages I get from the first run are below (I get them in the second run as well, but not in the final run).

MacBook-Pro-23:todo richard$ forge deploy
forge: 3 child task(s) errored
  scan: /Users/richard/dw/todo -> api, auth, todo-db, prometheus, search, tasks
  dependencies: api, auth, todo-db, prometheus, search, tasks ->
  service: api -> 1 child task(s) errored
    build: api:052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: api 052c07191e618f06ef20b8dc743c9377527c3883.ephemeral -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: auth -> 1 child task(s) errored
    build: auth:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: auth 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
  service: todo-db
    build: manifests /Users/richard/dw/todo/work/k8s/todo-db
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/todo-db -> OK
      storageclass "fast" created
      service "todo-db" created
      statefulset "todo-db" created
  service: prometheus
    build: pushed registry.hub.docker.com/richarddli/prometheus:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git
      manifests /Users/richard/dw/todo/work/k8s/prometheus
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/prometheus -> OK
      service "prometheus" created
      deployment "prometheus" created
      configmap "prometheus-config" created
  service: search
    build: manifests /Users/richard/dw/todo/work/k8s/search
    deploy: kubectl apply -f /Users/richard/dw/todo/work/k8s/search -> OK
      service "search" created
      deployment "search" created
  service: tasks -> 1 child task(s) errored
    build: tasks:2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
      push: 1 child task(s) errored
        push: tasks 2acf652dd67fc2396d5961298e8de6e3ca5cadc8.git -> 1 child task(s) errored
          CMD: command failed[1]: Error saving credentials: error storing credentials - err: exit status 1, out: `The specified item already exists in the keychai
n.`
@ark3

This comment has been minimized.

ark3 commented Aug 17, 2017

This looks like a Mac-specific issue with git and the macOS keychain credential helper, which Apple's Git uses by default, I believe. I can help you debug this. It would be useful to know what subcommands are being run that generate those errors.

><> fgrep -B1 keychain ~/.gitconfig
[credential]
  helper = osxkeychain
><> git credential-osxkeychain
usage: git credential-osxkeychain <get|store|erase>
@rhs

This comment has been minimized.

Contributor

rhs commented Aug 17, 2017

There should be a log of all the commands in /tmp/forge.log

@richarddli richarddli changed the title from Race condition? to Mac OS X: git credentials and keychain failure Aug 25, 2017

@richarddli

This comment has been minimized.

Contributor

richarddli commented Aug 25, 2017

`Step 1/4 : FROM datawire/ambassador-envoy:latest
---> f8e53d525e68
Step 2/4 : RUN apt-get update && apt-get -q install -y curl dnsutils
---> Using cache
---> f77fc9f9afaf
Step 3/4 : ENTRYPOINT /usr/local/bin/envoy
---> Using cache
---> 9adb9a03eef9
Step 4/4 : CMD -c /etc/envoy/envoy.json
---> Using cache
---> b07d55aae0ed
Successfully built b07d55aae0ed
Successfully tagged registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git
(0:00:01)
INFO forge[1].service[1].build[1].bake[1].docker-build[1]: RESULT -> None (0:00:01)
INFO forge[1].service[1].build[1].bake[1]: RESULT -> ['Dockerfile'] (0:00:03)
INFO forge[1].service[1].build[1].push[1]: START(api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1]: checking if api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git containers exist
INFO forge[1].service[1].build[1].push[1].applicator[1]: START((api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git, 'api', 'Dockerfile'))
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: START(docker, images, -q, registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: docker images -q registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc3... -> (in progress)
b07d55aae0ed

INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: docker images -q registry.hub.docker.com/richarddli/api:8587e3eb8701ef5dc3... -> b07d55aae0ed

INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1].CMD[1]: RESULT -> b07d55aae0ed
(0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].local_exists[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].remote_exists[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1].remote_exists[1]: RESULT -> False (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1].needs_push[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1].applicator[1]: RESULT -> True (0:00:00)
INFO forge[1].service[1].build[1].push[1]: pushing container Dockerfile
INFO forge[1].service[1].build[1].push[1].push[1]: START(api, 8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: START(docker, login, -u, richarddli, -p, , registry.hub.docker.com)
INFO forge[1].service[4].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[4].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[4].build[1]: RESULT -> ERROR (0:00:05)
INFO forge[1].service[4]: RESULT -> ERROR (0:00:05)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Login Succeeded

INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> Login Succeeded

INFO forge[1].service[5].build[1].push[1].push[1].CMD[1]: RESULT -> Login Succeeded
(0:00:02)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: START(docker, push, registry.hub.docker.com/richarddli/search:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]

INFO forge[1].service[3].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[3].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[3].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[3]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[2].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[2].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[2].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[2]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Error saving credentials: error storing credentials - err: exit status 1, out: The specified item already exists in the keychain.

INFO forge[1].service[1].build[1].push[1].push[1].CMD[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1].push[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1].push[1]: RESULT -> ERROR (0:00:02)
INFO forge[1].service[1].build[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[1]: RESULT -> ERROR (0:00:06)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> (in progress)
Login Succeeded

INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: docker login -u richarddli -p registry.hub.docker.com -> Login Succeeded

INFO forge[1].service[6].build[1].push[1].push[1].CMD[1]: RESULT -> Login Succeeded
(0:00:02)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[2]: START(docker, push, registry.hub.docker.com/richarddli/tasks:8587e3eb8701ef5dc369aeba1411f004ad60f08c.git)
INFO forge[1].service[6].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/tasks:8587e3eb8701ef5d... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/tasks]

INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]
9b01d945127f: Preparing

INFO forge[1].service[5].build[1].push[1].push[1].CMD[2]: docker push registry.hub.docker.com/richarddli/search:8587e3eb8701ef5... -> (in progress)
The push refers to a repository [registry.hub.docker.com/richarddli/search]
9b01d945127f: Preparing
77eaac6d0b1f: Preparing
`

@ark3

This comment has been minimized.

ark3 commented Aug 25, 2017

@richarddli Does the behavior change if you turn off "Securely store docker logins in macOS keychain" in Docker for Mac Preferences/General?

@ark3 ark3 changed the title from Mac OS X: git credentials and keychain failure to Mac OS X: (docker? git?) credentials and keychain failure Aug 25, 2017

@tristanpemble

This comment has been minimized.

Contributor

tristanpemble commented Aug 29, 2017

@ark3 I've seen this error come and go, and after disabling that setting in Docker for Mac it seemed to go away

@richarddli

This comment has been minimized.

Contributor

richarddli commented Aug 29, 2017

I haven't run Forge extensively since changing the setting but it does seem to make it go away.

@rhs

This comment has been minimized.

Contributor

rhs commented Aug 29, 2017

I found this issue that looks like it might be the same when googling this error: docker/for-mac#1540

It talks about removing "credsStore": "osxkeychain" from ~/.docker/config.json. Does anyone (or can someone with a mac check) if this is the same thing that the "Docker for Mac Preferences/General" workaround is tweaking?

If it is, then I might be able to fix this by launching docker with a custom config setup rather than letting it point to the default ~/.docker/config.json. This would have other UX implications though, e.g. any other config customizations someone might have made in that file wouldn't take effect for forge. I don't know how big a deal this is offhand. (I don't personally customize docker behavior all that much.) Alternatively I could detect the setting and/or the error message when this happens and warn/point to the workaround.

@richarddli

This comment has been minimized.

Contributor

richarddli commented Aug 29, 2017

So if the box is checked, the config.json file says:

{
  "auths" : {
    "registry.hub.docker.com" : {

    },
    "https://registry.hub.docker.com" : {

    }
  },
  "credsStore" : "osxkeychain"

If it is not checked, it reads:

{
	"auths": {
		"https://registry.hub.docker.com": {
			"auth": "cmljaGi89dfhjasdhjfhjdfhjadfhjhsdfdfU"
		},
		"registry.hub.docker.com": {
			"auth": "cmljaGi89dfhjasdhjfhjdfhjadfhjhsdfdfU"
		}
	}

(these are not my actual credentials)

@plombardi89

This comment has been minimized.

Contributor

plombardi89 commented Aug 29, 2017

I'm really not a fan of the idea of using a different Docker config file than the one used by default. It means I now need to maintain the logins to registries for Forge as well as when I am doing out of band Forge work involving push/pull from authorized or private registries.

richarddli added a commit that referenced this issue Aug 30, 2017

@richarddli

This comment has been minimized.

Contributor

richarddli commented Aug 30, 2017

I added documentation about configuring Docker for Mac as a temporary (and permanent?) solution.

@rhs rhs closed this Nov 7, 2017

@romualdr

This comment has been minimized.

romualdr commented Nov 24, 2017

MacOS Version 17.09.0-ce-mac35 (19611)

Docker login failed every time with "Securely store docker logins in macOS keychain"

error getting credentials - err: exit status 1, out: The user name or passphrase you entered is not correct.`

I turned the setting off and it worked as expected.

This setting is hell.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment