New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker errors in CoreOS 1029.0.0 #1246

Closed
brandonweeks opened this Issue Apr 27, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@brandonweeks

brandonweeks commented Apr 27, 2016

Using coreos-kubernetes to provision a cluster fails with the alpha channel/1029.0.0. Kubelet never starts, it seems that the Docker service has a bug introduced in 1029.0.0 as 1010.1.0 works with the same configuration.

Cluster created with kube-aws from coreos-kubernetes latest (f3f0af1) with releaseChannel: alpha and controllerInstanceType: m3.medium.

MOTD: https://gist.github.com/brandonweeks/02d1e1cdc3fbffd25739c81eb5c571c2
Output from journal -u docker --no-pager: https://gist.github.com/brandonweeks/57e65a91eb9635e5c7312e661b6523f2

@brandonweeks brandonweeks changed the title from Docker errors on to Docker errors in CoreOS 1029.0.0 Apr 27, 2016

@philips

This comment has been minimized.

Member

philips commented Apr 27, 2016

@brandonweeks yes, we had to pause the alpha rollout of 1029.0.0 because of this issue.

We are still trying to root cause it but it seems related to the systemd cgroups driver. If you remove the Environment="DOCKER_CGROUPS=--exec-opt native.cgroupdriver=systemd" from the docker.service file it will work (but some other things will break like kubelet, if you are using it).

We will keep this issue updated with details.

@mischief

This comment has been minimized.

mischief commented Apr 27, 2016

backporting opencontainers/runc#446 to docker 1.10.3 appears to fix the cpu shares issue, but i need to run kubernetes to be sure everything is ok.

@mischief

This comment has been minimized.

mischief commented Apr 27, 2016

fix for this should be in CoreOS alpha 1031.0.0, tomorrow.

@brandonweeks

This comment has been minimized.

brandonweeks commented Apr 27, 2016

Thank you @mischief!

@mischief

This comment has been minimized.

mischief commented Apr 28, 2016

1032.0.0 is out, so kube-aws should be fine now on alpha.

@mischief

This comment has been minimized.

mischief commented Apr 29, 2016

for posterity, a simple reproducer for this bug is:

docker run --rm -ti --cpu-shares=100 busybox
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment