Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

systemd ExecStop= Not Run #186

Closed
jbulchergist opened this issue Nov 5, 2014 · 9 comments
Closed

systemd ExecStop= Not Run #186

jbulchergist opened this issue Nov 5, 2014 · 9 comments

Comments

@jbulchergist
Copy link

I have unit files for various docker services, and I would like to run a docker container before they exit. This docker container manages DNS - there could be other conceivable uses for such "utility containers".

I have the appropriate Requires=docker.service and After=docker.service in the Unit file.

The ExecStop command is not run until after the docker.service stops. In fact, the first thing CoreOS seems to do when shutting down is to stop running docker containers. By the time systemd gets around to the unit file, the docker container is long gone - and in my case even the docker.service has already been shut down.

The implications of this issue are that no docker Unit files ever gracefully shut down as written, which seems like pretty significant flaw (even if it only affects a minority of users, since the containers themselves get shut down)..

My Unit files are below (a little worse for wear, as I have tried many different configurations)

registry-dns.service
-------------------
[Unit]
Description=Docker Registry DNS Registration
Before=registry.service registry-ui.service
After=network.target docker.service
Requires=network.target docker.service registry.service registry-ui.service
Conflicts=shutdown.target
[Service]
TimeoutStartSec=0
RemainAfterExit=yes
Type=oneshot
ExecStart=/usr/bin/docker run --rm argo:5000/nsupdate dns \
                add \
                --address 192.168.1.60 \
                --domain foo.local \
                --host argo \
                --nameserver 192.168.1.50 \
                --nameserver 192.168.1.51
ExecStop=/usr/bin/docker run --rm argo:5000/nsupdate dns \
                delete \
                --address 192.168.1.60 \
                --domain foo.local \
                --host argo \
                --nameserver 192.168.1.50 \
                --nameserver 192.168.1.51
[X-Fleet]
MachineOf=registry.service
[Install]
RequiredBy=registry.service registry-ui.service
Also=registry.service registry-ui.service
cat: registry-dns.service.requires: Is a directory
registry-ui.service
------------------
[Unit]
Description=Docker Registry UI
After=registry.service registry-dns.service
Requires=registry.service registry-dns.service
Conflicts=shutdown.target
[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start -a RegistryUI
ExecStop=/usr/bin/docker stop RegistryUI

[X-Fleet]
MachineOf=registry.service

[Install]
RequiredBy=registry.service registry-dns.service
Also=registry.service registry-dns.service
cat: registry-ui.service.requires: Is a directory
registry.service
-----------------
[Unit]
Description=Docker Registry
Requires=docker.service registry-dns.service registry-ui.service
After=docker.service registry-dns.service
Before=registry-ui.service
Conflicts=shutdown.target reboot.target

[Service]
TimeoutStartSec=0
Restart=on-failure
ExecStart=/usr/bin/docker start -a Registry
ExecStop=/bin/echo deleting dns
ExecStop=/usr/bin/docker run --rm argo:5000/nsupdate dns delete --address 192.168.1.60 --domain foo.local --host argo --nameserver 192.168.1.50 --nameserver 192.168.1.51
ExecStop=/bin/echo stopping registry
ExecStop=/usr/bin/docker stop Registry
ExecStop=/bin/echo stopped registry

[X-Fleet]
MachineID=coreos1

[Install]
WantedBy=multi-user.target
RequiredBy=registry-ui.service registry-dns.service docker.service
Also=registry-ui.service registry-dns.service
cat: registry.service.requires: Is a directory
@mailsquad
Copy link

Important for me also. Thanks.

@kaii-zen
Copy link

This affects me too. This is really important and I'm surprised not many people are +1ing this...
I'm running a Cassandra cluster in containers and it's important that the nodes decommission themselves gracefully otherwise the cluster ends up in a degraded state and new nodes cannot join until the dead nodes are cleared. This basically means that I cannot have CoreOS auto update which is a shame :(

@marineam
Copy link

We finally tracked this down, it is a flaw in the integration between systemd and docker. For every container docker registers a systemd 'scope' in order to create a cgroup for the container. systemd kills off all scopes first thing during the shut down sequence but the documentation claims this can be disabled by disabling DefaultDepdendencies for that scope unit. However it appears that it isn't actually possible to set that property making modifying the behavior currently impossible. So unless our understanding is wrong fixing that is the first step. Email asking about this was posted recently: http://lists.freedesktop.org/archives/systemd-devel/2014-December/026313.html

The second step after that will be to update docker to always set DefaultDepdendencies=false for each scope it creates. At the moment I don't know of a reasonable workaround.

@kaii-zen
Copy link

From my tinkering it seems the set-property doesn't work for properties inside the [Unit] section but only for properties in the [Scope], [Service], etc sections. I did have limited success doing the following:
Create /run/systemd/system/docker-<>.scope.d/50-DefaultDependencies.conf with the following contents:

[Unit]
DefaultDependencies=no

then reload systemd (by running 'systemctl reload-daemon')
After doing this and rebooting, my stop procedure would run as expected.
I tried to incorporate this into my unit file as a StartExecPost but reloading systemd in the middle of a unit execution prevents the unit from changing state from activating to activated.
I see @philips's email is not getting a response from the systemd folks but my test shows that setting DefaultDependencies=no does indeed work and we only need Docker to set it.

@philips
Copy link

philips commented Feb 4, 2015

Update on this, it is fixed in systemd upstream and I am patching Docker's libcontainer to fix it too: docker-archive/libcontainer#359

@mailsquad
Copy link

Great work, thanks!

@kaii-zen
Copy link

kaii-zen commented Feb 4, 2015

Awesome. Any idea when the changes will be released in coreos?

@vcaputo
Copy link

vcaputo commented Feb 5, 2015

@kreisys fixed in the next alpha release

@crawford
Copy link
Contributor

Fixed in 584.0.0.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants