New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kops does not use DOCKER_OPTS or /etc/sysconfig/docker in docker.service #1731

Closed
imsky opened this Issue Jan 31, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@imsky
Copy link

imsky commented Jan 31, 2017

i'm using kops 1.5 to tweak the storage driver on a cluster and while /etc/sysconfig/docker receives the correct flags, docker.service does not use that file as an EnvironmentFile and does not append $DOCKER_OPTS to the main command. this leads to dockerd using devicemapper as the storage driver.

@chrislovecnm

This comment has been minimized.

Copy link
Member

chrislovecnm commented Feb 2, 2017

What would you expect?

@imsky

This comment has been minimized.

Copy link
Author

imsky commented Feb 3, 2017

i'd expect that the DOCKER_OPTS are set in the systemd docker service so that changing the storage driver in kops cluster settings actually has an effect

@justinsb justinsb added this to the 1.5.0 milestone Feb 5, 2017

@justinsb justinsb added the P1 label Feb 5, 2017

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Feb 5, 2017

This is fixed in kops 1.5.0, which should release today.

For example, I set:

  docker:
    storage: overlay2

and docker was run with /usr/bin/dockerd --ip-masq=false --iptables=false --log-level=warn --storage-driver=overlay2

(note this is only an example, as overlay2 is not officially supported with k8s 1.5)

Closing; please reopen if not fixed in 1.5.0.

@justinsb justinsb closed this Feb 5, 2017

@imsky

This comment has been minimized.

Copy link
Author

imsky commented Feb 7, 2017

@justinsb unfortunately this is not fixed. the systemd service file is correctly generated and the docker version is set correctly, but the storage driver still defaults to devicemapper. restarting the docker service via systemd fixes the issue, but this is something kops should do already.

@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Mar 10, 2017

Ouch - sorry @imsky

Can you confirm the AMI you are running & the kops version please?

Edit, from slack: "we're on 1.5.1, ami is k8s-1.4-debian-jessie-amd64-hvm-ebs-2016-10-21 (ami-4bb3e05c)"

justinsb added a commit to justinsb/kops that referenced this issue Mar 11, 2017

service task: consider changes to the systemd manifest itself
We do smart service restarting: we restart the service if a dependency
changed after the running service.  However, we were not considering the
service manifest itself in the calculation, which was an error.

The bug only exposed itself though when we downloaded and updated
docker, e.g. when running k8s 1.5 with a 1.4 image.  We would write the
env file early (no dependencies), then we would download docker and
install it, and then we would write the service manifest.  But if docker
had been started during this interval (e.g. by protokube), then we would
see that docker had been started after the dependencies (the env file),
and not restart it.  When we consider the manifest file also, things
work as intended.

Fix kubernetes#1731
@justinsb

This comment has been minimized.

Copy link
Member

justinsb commented Mar 11, 2017

I think #2102 will fix it @imsky - see the comment for an explanation.

As a simple workaround, if you update the AMI to 1.5, it should "just work". kops upgrade cluster should recommend an image upgrade, I believe; you can either apply it manually or accept its recommendations (it'll probably recommend a k8s upgrade also though).

justinsb added a commit to justinsb/kops that referenced this issue Mar 13, 2017

service task: consider changes to the systemd manifest itself
We do smart service restarting: we restart the service if a dependency
changed after the running service.  However, we were not considering the
service manifest itself in the calculation, which was an error.

The bug only exposed itself though when we downloaded and updated
docker, e.g. when running k8s 1.5 with a 1.4 image.  We would write the
env file early (no dependencies), then we would download docker and
install it, and then we would write the service manifest.  But if docker
had been started during this interval (e.g. by protokube), then we would
see that docker had been started after the dependencies (the env file),
and not restart it.  When we consider the manifest file also, things
work as intended.

Fix kubernetes#1731
@imsky

This comment has been minimized.

Copy link
Author

imsky commented Mar 16, 2017

yes @justinsb - after upgrading AMI to 1.5 it worked out of the box

thanks for following up on this

justinsb added a commit to justinsb/kops that referenced this issue Mar 19, 2017

service task: consider changes to the systemd manifest itself
We do smart service restarting: we restart the service if a dependency
changed after the running service.  However, we were not considering the
service manifest itself in the calculation, which was an error.

The bug only exposed itself though when we downloaded and updated
docker, e.g. when running k8s 1.5 with a 1.4 image.  We would write the
env file early (no dependencies), then we would download docker and
install it, and then we would write the service manifest.  But if docker
had been started during this interval (e.g. by protokube), then we would
see that docker had been started after the dependencies (the env file),
and not restart it.  When we consider the manifest file also, things
work as intended.

Fix kubernetes#1731

justinsb added a commit to justinsb/kops that referenced this issue Mar 21, 2017

service task: consider changes to the systemd manifest itself
We do smart service restarting: we restart the service if a dependency
changed after the running service.  However, we were not considering the
service manifest itself in the calculation, which was an error.

The bug only exposed itself though when we downloaded and updated
docker, e.g. when running k8s 1.5 with a 1.4 image.  We would write the
env file early (no dependencies), then we would download docker and
install it, and then we would write the service manifest.  But if docker
had been started during this interval (e.g. by protokube), then we would
see that docker had been started after the dependencies (the env file),
and not restart it.  When we consider the manifest file also, things
work as intended.

Fix kubernetes#1731

justinsb added a commit to justinsb/kops that referenced this issue Mar 22, 2017

service task: consider changes to the systemd manifest itself
We do smart service restarting: we restart the service if a dependency
changed after the running service.  However, we were not considering the
service manifest itself in the calculation, which was an error.

The bug only exposed itself though when we downloaded and updated
docker, e.g. when running k8s 1.5 with a 1.4 image.  We would write the
env file early (no dependencies), then we would download docker and
install it, and then we would write the service manifest.  But if docker
had been started during this interval (e.g. by protokube), then we would
see that docker had been started after the dependencies (the env file),
and not restart it.  When we consider the manifest file also, things
work as intended.

Fix kubernetes#1731
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment