no such process error when container state is inconsistent

[crosbymichael]

Error that @discordianfish received when upgrading. A container's state file says that the container is running but there is no pid and you cannot make docker kill the container because it is returning an error that is cannot find the process.

[wyaeld]

Pretty certain this has happened to me recently.

ubuntu@sot-prod:~$ sudo docker ps
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS                                                   NAMES
3904cca20f4a        datacom/sot-rails:latest       /sbin/my_init          6 days ago          Up 6 days           443/tcp, 0.0.0.0:3003->80/tcp                           thirsty_colden      
fd839d4c8d39        datacom/sot-wordpress:backup   /bin/bash /start.sh    7 days ago          Up 7 days           0.0.0.0:2222->22/tcp, 3306/tcp, 0.0.0.0:10080->80/tcp   stupefied_ptolemy   
856ee992e1bb        wyaeld/postgres:latest         /usr/lib/postgresql/   7 days ago          Up 7 days           0.0.0.0:5432->5432/tcp                                  clever_kirch        
e934e776e86b        wyaeld/postgres:latest         /usr/lib/postgresql/   11 days ago         Up 7 days                                                                   high_lovelace       
d15e433d5820        datacom/sot-wordpress:1.1      /bin/bash /start.sh    2 weeks ago         Up 7 days                                                                   lonely_newton       
ubuntu@sot-prod:~$ sudo docker kill e934e776e86b
Error: Cannot kill container e934e776e86b: no such process
2014/05/21 09:03:43 Error: failed to kill one or more containers
ubuntu@sot-prod:~$ sudo docker kill d15e433d5820
Error: Cannot kill container d15e433d5820: no such process
2014/05/21 09:04:06 Error: failed to kill one or more containers

I've been trying to work how how to manually remove them. In this case it was caused by not upgrading, but stopping docker, moving everying in /var/lib/docker onto an openstack volume, and then updating the config and restarting the daemon. Most of the containers came back, but these 2 - both of which have DBs in them, did not.

[cyphar]

I also have seen this. It looks like somewhere the state isn't being updated (for me, it usually occurs after hard reboots) or reset. It might have something to do with how docker restarts the previously running containers on startup (and it doesn't check if they actually are running before tagging them as "running" in the daemon).

[sheerun]

I also experienced this. It happened after server restart.

[discordianfish]

@wyaeld @cyphar @sheerun: Have you found any way to reproduce this? I haven't so far :-/

[sheerun]

@discordianfish It happened for dokku application instance, so I guess it has to do with the way they restart container to bind previously assigned ports. Maybe it's some kind of race condition. Maybe to recreate you need to kill container in upstart script, and restart it, or something.

[discordianfish]

@sheerun Could you try to reproduce it and describe how? That would help a lot!

[sheerun]

Sorry, I already uninstalled dokku on server, and I cannot recreate in vagrant.

[cyphar]

@discordianfish I can reproduce on ubuntu 14.04 (server). Set up several docker instances, then reboot the server. On restart, when docker starts automatically, there will be several "zombie" containers. The only way to remove them is to restart the docker daemon and then kill them (since they become un-zombified after a clean restart of the docker daemon).

[erikh]

I'm seeing something slightly related to this:

1. start some busybox processes that won't/can't be killed with SIGTERM:
   • docker run -d busybox sh -c 'fart() { sleep 10000000; }; trap fart TERM INT; while true; do sleep 1; done'
2. sudo shutdown -r now

Note that sudo pkill docker and sudo pkill -9 docker will not trigger this. I'm thinking it may be related to how docker is shutdown in relationship to the container's processes.

[crosbymichael]

ping @erikh. Any update on this?