Error reading X509 key pair (...) crypto/tls: private key does not match public key

[peter-leonov]

Description of problem:

Weird error log messages from a running Docker daemon.

docker version:

Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

docker info:

Containers: 6
Images: 175
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 187
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-52-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 1.955 GiB
Name: uk
ID: VPTY:47IV:KZWE:JAOO:CVAC:QKX2:FD5K:LJE3:OELB:4HOJ:ID7O:RJR5
WARNING: No swap limit support
Labels:
 provider=digitalocean

uname -a:

Linux uk 3.13.0-52-generic #85-Ubuntu SMP Wed Apr 29 16:44:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

A DigitalOcean droplet created and provided by docker-machine v0.2.0 on 22.06.2015
Droplet details: 2GB Ram 30GB SSD Disk Amsterdam 3 Ubuntu 14.04 x64

How reproducible:

Stop docker daemon using Upstart with stop docker or wait for the daemon to unexpectedly shutdown itself (a subject to another issue), then check out /var/log/upstart/docker.log

Steps to Reproduce:

1. sudo stop docker
2. sudo start docker
3. grep ERRO /var/log/upstart/docker.log

Actual Results:

Lots of weird errors reported including few of this:

ERRO[0000] ServeAPI error: Error reading X509 key pair (/etc/docker/server.pem, /etc/docker/server-key.pem): "crypto/tls: private key does not match public key". Make sure the key is encrypted.

Expected Results:

No errors :)

Additional info:

Full log from /var/log/upstart/docker.log: https://gist.github.com/kung-fu-tzu/49cb28698a93064d79cb

[phemmer]

Where did that /etc/docker/server.pem and /etc/docker/server-key.pem come from? As the error indicates, the private key (server-key.pem) does not correspond to the public key (server.pem).

[peter-leonov]

It's a fresh DO droplet created and provisioned by docker-machine. As I believe docker-machine uses the official way to install Docker: curl docker.io/bla-bla | sh.

[phemmer]

I would recommend opening a ticket over on the docker machine issue tracker. It is the one responsible for generating those key files.

[peter-leonov]

Indeed the docker machine is responsible for the initial setup, but the precise problem is the daemon worked for a few days, then logged these error messages, and then worked another handful of days without a notice.

Do you still think I need to duplicate this issue to the docker machine issue tracker?

[phemmer]

If you're saying the docker daemon randomly restarts, throws this error a few times when trying to start, and then all of a sudden starts up fine, all without human interaction, then you likely have something running on the box which is causing these problems. Likely this thing that is running (perhaps the docker machine provisioning process) is regenerating the server keys, and every time docker fails to start, it's because it reads the keys in the middle of them being being regenerated.
This would be easy to confirm, just do an ls -l on those files, and see if the timestamp is changing.

But in any case, I don't think docker is the issue here (the code generating that error is part of the golang standard library. The chances of it being broken are pretty low). I can't say for sure it's docker machine, but it is a possibility.

[peter-leonov]

Looks like it is possible the certs were re-generated:

root@uk:~# ls -l /etc/docker/server*
-rw-r--r-- 1 root root 1675 Jun 29 16:34 /etc/docker/server-key.pem
-rw-r--r-- 1 root root 1086 Jun 29 16:33 /etc/docker/server.pem

Timestamp is indeed changed a week after the droplet was created. Adding an issue to the machine tracker too. Thanks!

[icecrime]

Closing this as it seems to be tracked in Machine, and doesn't indeed point to a Docker Engine issue. Thanks!