Use client cert to access secure private registry

[hobti01]

Expected behavior

docker login my.secure.registry
Username: myusername
Password:
Login Succeeded

Actual behavior

docker login my.secure.registry
Username: myusername
Password:
Error response from daemon: Get https://my.secure.registry/v1/users/: remote error: tls: handshake failure

Information

Diagnostic ID: 6A4673D8-43E5-4A49-8E31-CB17E771C8C3
Docker for Mac: version: 1.13.1 (94675c5a7)
macOS: version 10.12.3 (build: 16D32)
logs: /tmp/6A4673D8-43E5-4A49-8E31-CB17E771C8C3/20170222-153031.tar.gz
[OK] vmnetd
[OK] dns
[OK] driver.amd64-linux
[OK] virtualization VT-X
[OK] app
[OK] moby
[OK] system
[OK] moby-syslog
[OK] db
[OK] env
[OK] virtualization kern.hv_support
[OK] slirp
[OK] osxfs
[OK] moby-console
[OK] logs
[OK] docker-cli
[OK] menubar
[OK] disk

Steps to reproduce the behavior

1. Implement docker registry protected by TLS on port 443 and requiring client certificate
2. Add CA cert for registry to Mac OS keychain
3. Attempt login

According to the Linux documentation the client cert and key can be stored in /etc/docker/certs.d and this is confirmed to work on Linux.

/etc/docker/certs.d/my.secure.registry/client.cert
/etc/docker/certs.d/my.secure.registry/client.key

With Docker for Mac it might be presumed (there is no documentation regarding client certificates for Docker for Mac) that ~/.docker could be used for the same purpose as /etc/docker but the following does not work:

~/.docker/certs.d/my.secure.registry/client.cert
~/.docker/certs.d/my.secure.registry/client.key

How can a client cert be used on Docker for Mac?

[ijc]

Hi @hobti01 thank you for your report. Adding the key to the MacOS keychain is supposed to be sufficient according to https://docs.docker.com/docker-for-mac/faqs/#how-do-i-add-custom-ca-certificates, although I've not tried this myself.

According to the docs you should need a step 2b: restart docker 4 mac.

For step 2 what command did you use to add the cert to the keychain?

[hobti01]

When you say "key", which key do you mean? I am explicitly asking about the client cert and key, not the server cert or ca cert. You have linked to the section on ca certificates, which is not relevant to client certificates. :(

Even if the client cert would be used from the keychain, there must be a method to specify which client cert to use to connect to this registry.

[ijc]

Thanks for correcting me, I was indeed confusing server certs with client certs.

I've now raised a feature request for this internally.

[hobti01]

Thank you! I'm happy to provide additional details. I have also created a related issue moby/moby#31295

[matthewbarr]

We actually need this as well, and would love to pull the client cert & key out of the keychain. Curl on the Mac actually implements this.

It might be possible to use the login subcommand to specify which client cert to use for a specific host.

For now, we actually copy the cert / key to a directory in the xhyve VM, via the git repo in ~/Library/Containers/com.docker.docker/Data/database/

and git reset --hard

then adding to : com.docker.driver.amd64-linux/etc/docker/certs.d/$HOSTNAME/

Just like you would on the linux system.

[ijc]

@matthewbarr you would also need to git add + git commit those new files to make things visible I think?

[matthewbarr]

I didn't pull the full info in, sorry. Yes, you're correct. And then restart the xhyve VM, too.

We don't like storing private key's unencrypted on disk, but there's no other way for now.

We will probably start deleting the files after we do the commit, so they aren't left on disk in quite so obvious a manner.

[ijc]

And then restart the xhyve VM, too.

It's supposed to autorestart when you do the git commit, is this not happening for you?

[hobti01]

@matthewbarr Could you help me understand the steps for this workaround? I see that you've been part of the discussion in the forums where there are other workarounds, but not as you describe. I think you are doing something like this?

Create registry directories within your docker dir:
~/.docker/certs.d/my-registry.hostname.com:port

then:

VM_DIR=com.docker.driver.amd64-linux/etc/docker
cd ~/Library/Containers/com.docker.docker/Data/database
git reset --hard
mkdir -p ${VM_DIR}
rm -rf ${VM_DIR}/certs.d
cp -R -L ~/.docker/certs.d ${VM_DIR}
git add .
git commit -m "Add client certs"
rm -rf com.docker.driver.amd64-linux
git add .

echo "You must restart Docker!"

It seems to me that everything is left in more-or-less the state it was in before the manual addition, but it's unclear to me what the git status should be. Could this be improved?

@ijc25 git commit within a shell would cause a VM restart, but within a script did not restart the VM

[ijc]

@hobti01 it turns out it depends on which path you touch, only certain paths are explicitly watched and cause a reboot. So for example:

$ cat ~/enable-docker-debug.sh 
#!/bin/bash
set -e


VM_DIR=com.docker.driver.amd64-linux/etc/docker
cd ~/Library/Containers/com.docker.docker/Data/database
git reset --hard
echo '{"debug": true}' > $VM_DIR/daemon.json
git add $VM_DIR/daemon.json
git commit -m "Enable engine debug"

Running that script does do a reboot. However you are correct, certs.d is not one of the watched paths and therefore doesn't cause a reboot so it needs to be done by hand, sorry for the earlier misinformation.

[matthewbarr]

Is there a safe file that can cause the reboot? or another safe way to ensure that the reboot happens? I'd like to be able to script that.

(And I'll post our new script soon.)

[ijc]

I don't see anything which looks like it could be idempotently prodded to force a change. You could add/remove a blank line from the end of several files though, com.docker.driver.amd64-linux/etc/docker/daemon.json looks like it would be safe enough to do that to.

[matthewbarr]

We store our client certs in the keychain, and you can extract the public half easily. You can't get the key from the keychain, though. We happen to store it in ~/.certs, encrypted, with the password stored in the keychain, though.

If you have the cert / key pair somewhere else, you can just replace the middle code section.

#!/bin/bash
DOCKER_HOST=devprod.docker-dev.example.com
DOCKER_CERT_LOCATION=~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/etc/docker/certs.d/$DOCKER_HOST

cd ~/Library/Containers/com.docker.docker/Data/database/
git reset --hard

cd com.docker.driver.amd64-linux/etc/docker/
mkdir -p certs.d/$DOCKER_HOST


#COPY CERTS into ~/Library/Containers/com.docker.docker/Data/database/com.docker.driver.amd64-linux/etc/docker/....>
# they must be named foo.cert & foo.key,  no passwords.
# Retrieve the public half of the client cert from your keychain
security get-identity-preference -s '*.example.com' -p  > $DOCKER_CERT_LOCATION/user.cert

# Find latest private key in ~/.certs 
key=`ls -t  ~/.certs/*.key| head -n1`
echo $key

# find password for private key in keychain, save non-encrypted version to $DOCKER_CERT_LOCATION
security find-generic-password -w -s `basename $key .key` | openssl rsa -in $key -passin stdin -out $DOCKER_CERT_LOCATION/user.key


#add line to daemon.json to trigger an auto restart of docker VM, when committed to git.
echo >> daemon.json
git add daemon.json

git add certs.d/
git commit -m "add certs to docker"

There's no real reason to delete the directories from disk, unless you'd like to ensure they aren't stored on disk in an unencrypted form which can be found with a find ~ |grep key..
-- Which you might want to do, and I'll probably add soon, after I finish developing the user scripts.

[matthewbarr]

This is all work we're doing to roll out Docker for the Mac to Akamai developers. I'm going to probably try something similar for Windows, too, but that's another kettle of fish.

[hobti01]

@matthewbarr Thanks for your script. A Docker for Windows solution would also be useful unless a cross-platform solution is implemented. I'm happy to use the workaround for now, but would like to see Docker for Mac enhanced to support this in a more natural way.

[noderunner]

The latest version of Edge: Docker Community Edition 17.06.0-rc1-ce-mac13, 2017-06-01 (edge), supposedly fixes this issue. Unfortunately, I haven't been able to find any documentation on how a user is supposed to configure client certs with this update. Does anyone know how this is supposed to work? It looks like the update is an inclusion of this other project: https://github.com/docker/docker-credential-helpers, but I can't find any examples on how I'm supposed to configure my client key/cert.

Update: It looks like the work-around method described earlier of adding your certs into the xhyve VM no longer works with this update to Edge. I'm gonna have to revert to Stable to have a working system for now.

[matthewbarr]

@ijc - any where we can look for documentation on how to test this feature, since it's in the RC's for 17.06? It's broken the existing functionality, and the RC's are moving along. I'm afraid y'all will release it without having a resolution, and we'll be stuck w/ developers downloading stable and not being able to work at all.

We actually don't use passwords on our docker registry, just the client cert which provides the username, which is going to be interesting, and possibly require some changes on our registry side. Docker login does require you to send a username/password, which means we'll have to strip the password out, somehow. I'd really like to make sure to get some testing and fix time before the 17.06 release, since it'll go to stable, vs edge.

[matthewbarr]

In particular: We need to know:

1. How does one select which client cert should be sent a request?

2. How can one debug what's being sent?

[matthewbarr]

In particular: We need to know:

1. How does one select which client cert should be sent a request?

2. How can one debug what's being sent?

[ebriney]

@matthewbarr : with 17.06, you don't have to push your certs with git commands anymore, we copy
certificates from your mac folder ~/.docker/certs.d in the database when starting the app.

[matthewbarr]

Thanks! I'd actually intended to have it pull from the keychain, but that's not what this issue requested, so.. I'll go open a proper request for that :) This issue was the result of other conversation, which I think specified that, but.. fair is fair!

Thanks!

[londoncalling]

@matthewbarr we link to this Blog post on Adding Self-signed Registry Certs to Docker & Docker for Mac from the docs FAQ (How do I add custom CA certificates?). Is this what you are looking for?

[matthewbarr]

Nope - this is all about the client certs. I'd hoped to be able to use the client certs stored in the keychain, but that's not what this issue said :) And it'd be much harder, since we don't have a good mechanism for specifying the host to use a specific keychain client cert.

I just wasn't expecting to actually being able to use the Mac .docker/certs.d directory. It's obvious, now, but after figuring out the various workarounds, I wasn't expecting it.

[londoncalling]

@matthewbarr Thanks, I'll add to the docs that this is now possible.

[guillaumerose]

Closed in favor of #1757

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked