docker stop - SIGTERM/SIGINT aware processes exit only with SIGKILL

[unclejack]

This issue is rather strange. Processes which handle SIGTERM and SIGINT properly exit with exit code 137. This is the exit code shown in docker ps and this is the exit code printed by docker wait. Exit code 137 is usually the exit code one gets after sending SIGKILL to a process.

In the case of jpetazzo/pgsql, it exits cleanly within the container, but the exit code of the container is 137

PostgreSQL stand-alone backend 9.1.3
backend> backend> 2013-06-28 21:37:16 UTC LOG:  database system was shut down at 2013-06-28 21:37:16 UTC
2013-06-28 21:37:16 UTC LOG:  autovacuum launcher started
2013-06-28 21:37:16 UTC LOG:  database system is ready to accept connections

Not being able to stop some processes launched in the containers via SIGTERM/SIGINT (even though they do handle those signals properly when running on the host) and not being able to get the actual exit code makes it very difficult to monitor processes running under docker containers with a process monitoring solution.

Processes which usually exit with exit code 0 after a SIGTERM/SIGINT exit with exit code 137. This is detected as an error.

Based on the above, we either have 2 issues or one issue causing two problems:
1 SIGTERM isn't always passed on properly to the process running in the container
Docker stop has to reach the timeout and kill the container with SIGKILL.
This happens even with processes which handle SIGTERM / SIGINT properly on the host.

2 The real exit code of the process running in the container isn't shown in docker ps
This seems to happen when launching something from a shell script within another shell and in many other situations.
The only case in which I was able to get the real exit code of the container & make it handle SIGTERM is when launching pgsql by hand (directly running su postgres sh -c "....", without using the shell script) from the pgsql docker image.

update:
Here are some examples:

$ docker run -d jpetazzo/pgsql su postgres sh -c "/usr/lib/postgresql/9.1/bin/postgres           -D  /var/lib/postgresql/9.1/main  -c config_file=/etc/postgresql/9.1/main/postgresql.conf  -c listen_addresses=*"
c575abf843b6
$ docker stop c575abf843b6
$ docker ps -l
ID                  IMAGE                COMMAND                CREATED             STATUS              PORTS
c575abf843b6        jpetazzo/pgsql:9.1   su postgres sh -c /u   31 seconds ago      Exit 0
docker logs c575abf843b6
2013-06-28 22:16:59 UTC LOG:  database system was interrupted; last known up at 2013-05-01 07:08:30 UTC
2013-06-28 22:16:59 UTC LOG:  database system was not properly shut down; automatic recovery in progress
2013-06-28 22:16:59 UTC LOG:  record with zero length at 0/16BC7E0
2013-06-28 22:16:59 UTC LOG:  redo is not required
2013-06-28 22:16:59 UTC LOG:  autovacuum launcher started
2013-06-28 22:16:59 UTC LOG:  database system is ready to accept connections

Session terminated, terminating shell...2013-06-28 22:17:14 UTC LOG:  received smart shutdown request
2013-06-28 22:17:14 UTC LOG:  autovacuum launcher shutting down
2013-06-28 22:17:14 UTC LOG:  shutting down
2013-06-28 22:17:14 UTC LOG:  database system is shut down
 ...terminated.
# this is OK

$ docker run -d jpetazzo/pgsql /init
56242ec60f02
$ docker stop 56242ec60f02
$ docker ps -l
ID                  IMAGE                COMMAND             CREATED             STATUS              PORTS
56242ec60f02        jpetazzo/pgsql:9.1   /init               3 minutes ago       Exit 137
$ docker logs 56242ec60f02

PostgreSQL stand-alone backend 9.1.3
backend> backend> 2013-06-28 22:18:52 UTC LOG:  database system was shut down at 2013-06-28 22:18:51 UTC
2013-06-28 22:18:52 UTC LOG:  database system is ready to accept connections
2013-06-28 22:18:52 UTC LOG:  autovacuum launcher started
# not OK

/cc @shykes @jpetazzo @creack

[crosbymichael]

@unclejack I merged your PR today, is this still relevant?

[unclejack]

@crosbymichael Yes, this is still a problem. I'll close it since it requires further investigation and it's not immediately actionable.

[abourget]

Any news about this issue ? I've got postgresql that doesn't close cleanly with docker stop.

I'd certainly prefer to shutdown the DB the right way, instead of cutting it off altogether.

[jpetazzo]

The right solution might be #2100 (shutdown hooks), what do you think?

[mainframe]

Same problem with mysql (mariadb-galera cluster) - unclean (forced) shutdown with docker stop causes cluster to split brain (when stopping 2 nodes out of 3 cluster nodes). Shutdown hook could be a solution - at least better than nothing atm.

[mainframe]

Well - with 1.3 exec feature there is a possibility for a workaround for time being - just executing before docker stop cmd:

docker exec -it <ct-name> <process-graceful-stop-script-or-cmd>

Worked for me - but shutdown hook would be more proper way imho.

[mhsmith]

@abourget: postgres may not shut down in a timely manner in response to docker stop: see http://serverfault.com/questions/624903/stop-postgresql-container