Docker errors in CoreOS 1029.0.0

[brandonweeks]

Using coreos-kubernetes to provision a cluster fails with the alpha channel/1029.0.0. Kubelet never starts, it seems that the Docker service has a bug introduced in 1029.0.0 as 1010.1.0 works with the same configuration.

Cluster created with kube-aws from coreos-kubernetes latest (f3f0af1) with releaseChannel: alpha and controllerInstanceType: m3.medium.

MOTD: https://gist.github.com/brandonweeks/02d1e1cdc3fbffd25739c81eb5c571c2
Output from journal -u docker --no-pager: https://gist.github.com/brandonweeks/57e65a91eb9635e5c7312e661b6523f2

[philips]

@brandonweeks yes, we had to pause the alpha rollout of 1029.0.0 because of this issue.

We are still trying to root cause it but it seems related to the systemd cgroups driver. If you remove the Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd" from the docker.service file it will work (but some other things will break like kubelet, if you are using it).

We will keep this issue updated with details.

[mischief]

backporting opencontainers/runc#446 to docker 1.10.3 appears to fix the cpu shares issue, but i need to run kubernetes to be sure everything is ok.

[mischief]

fix for this should be in CoreOS alpha 1031.0.0, tomorrow.

[brandonweeks]

Thank you @mischief!

[mischief]

1032.0.0 is out, so kube-aws should be fine now on alpha.

[mischief]

for posterity, a simple reproducer for this bug is:

docker run --rm -ti --cpu-shares=100 busybox