Discovering and reattaching to running containers

[tiborvass]

We'd like the Docker daemon to be able to die and discover its containers upon restarting.

What are good implementations for achieving this? Having the daemon set some ID (caller ID?) in the containers' state?

It would also need to have a way of reattaching to standard streams of those running containers.

We also need to keep in mind the scenario when multiple docker daemons (or equivalent) spawn their own sets of containers; there shouldn't be interference between those instances.

[dqminh]

Reattaching stream is kinda hard. I think there're some hacks involving ptrace to let you reattach the stdin/stdout, but im not sure

What do you think about runc exposing a set of APIs ( HTTP ? ) that allows the monitor ( docker ) in this case to monitor and attach to the container stream ? The API is probably mirrors docker streaming API, but maybe with some additional healthchecks to make sure that runc is running properly.

Something like this:

• docker starts a runc process
• runc creates its own unix socket and register that socket with docker
• docker attaches to runc and tell it to start the actual process, streaming stdin/stdout back
• if the daemon restarts, runc processes reparented to INIT, but keeps running

Running one unix socket per container is also helpful in terms of discovery. A docker engine can reserve its /var/run/docker/engine-id/ directory and symlink the sockets of its managed container there for discovery purpose. So when the daemon restarts, it can find the current runc processes there.

Some thoughts:

• What do we do with container's stdin/stdout/stderr when the daemon dies ? Ideally i would want the container to keep running, which means stdout/stderr need to go to somewhere ?
• Overhead of APIs vs current way of passing file fds directly ?

[chenchun]

I agree with @dqminh . Actually I have made some progress on implementing docker hot upgrade. If you are interested, see https://github.com/chenchun/docker/commits/upgrades.

When I started working on this, I haven't head about runc, so I create an external monitor process per container like the design proposal of moby/moby#13884. The external monitor process is responsible for monitoring the container, notifying daemon of container starting or exiting, and providing extra streaming APIs such as /containers/{name:.*}/resize and /containers/{name:.*}/attach for attaching to the container. It manages to offer these APIs by listening to its own unix socket /var/run/${containerid}.sock. For the compatibility reason, docker daemon returns http.StatusMovedPermanently and Location: /var/run/${containerid}.sock to tell docker clients to redirect its streaming requests to the external monitor. With the below change container's stdin/stdout/stderr is not affected when the daemon dies and I assume if the external monitor process dies, docker daemon will kill the container process (This is not implemented yet in my branch).

[vbatts]

Initially, these are two different problems to solve. There could be a layout for discovering running containers, like a /var/run/containers structure that has pid, ns and otherwise information, that can be investigated to determine the pids running still.

Reattaching is surely more tricky, unless there is a per-container supervisor or init.

[mudverma]

@chenchun Good to know that you went ahead with design akin to our Proposal A... Just a question for you, how about the parent child relationship between the daemon and the containers? Do you daemonize Monitor+Container?

[chenchun]

@mudverma Actually, I'm doing Proposal B right now. The relationship between the daemon and the containers is the same with the graph in Proposal B.

[mudverma]

@chenchun thanks .. I pulled your code and ran it. When I killed the daemon, the containers were killed by the daemon on restart ? Also, I don't see any monitor process either in between the daemon and the containers. Am I missing something?

| `-docker-1.8.0-de,32642 -D -d

| |-sh,32706 -c while true; do echo hello world; sleep 1; done
| | `-sleep,319 1

| |-sh,32752 -c while true; do echo hello world; sleep 1; done
| | `-sleep,318 1
| |-{docker-1.8.0-de},32643
| |-{docker-1.8.0-de},32644
| |-{docker-1.8.0-de},32645

[chenchun]

@mudverma, How did you killed the daemon? If you are using service docker stop, please kill it directly with kill -9 $dockerpid cause it may be something else which killed the whole process group.

[chenchun]

@mudverma you have to copy the newly compiled docker to /usr/bin/docker, see https://github.com/chenchun/docker/blob/upgrades/docker/docker_monitor.go#L17. Sorry about this, I tried to implement a prototype quickly, so I made some hard codings which I'll address later.

Before, I started a ubuntu container

chenchun@chenchun-T450:~/project/go/src/github.com/docker/docker$ docker run -it ubuntu bash
INFO[0001] redirect to [unix /var/run/8efbc15d087eef9189e21fc6d181e28e6d8c084af00c9ae891488c8b8328a12b.sock] 
root@8efbc15d087e:/# ls /
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

//in another screen
root@chenchun-T450:/home/chenchun/project/go/src/github.com/docker/docker# ps -ef --forest | grep "7056\|7381" | grep -v grep
root      7056     1  0 09:30 ?        00:00:00 /usr/bin/docker -d --bip 192.168.1.0/16 -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock
root      7381  7056  0 09:31 ?        00:00:00  \_ docker_monitor 8efbc15d087eef9189e21fc6d181e28e6d8c084af00c9ae891488c8b8328a12b /var/lib/docker/containers
root      7387  7381  0 09:31 pts/33   00:00:00      \_ bash

Now kill the daemon directly. Interactive mode is not effected. (docker daemon restarts due to ubuntu upstart)

root@chenchun-T450:/home/chenchun/project/go/src/github.com/docker/docker# kill -9 `cat /var/run/docker.pid`
root@chenchun-T450:/home/chenchun/project/go/src/github.com/docker/docker# ps -ef --forest | grep "`cat /var/run/docker.pid`\|7381" | grep -v grep
root      7381     1  0 09:31 ?        00:00:00 docker_monitor 8efbc15d087eef9189e21fc6d181e28e6d8c084af00c9ae891488c8b8328a12b /var/lib/docker/containers
root      7387  7381  0 09:31 pts/33   00:00:00  \_ bash
root      8451     1  0 09:35 ?        00:00:00 /usr/bin/docker -d --bip 192.168.1.0/16 -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

//in the screen which starts ubuntu container
root@8efbc15d087e:/# ls /
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

[tonistiigi]

We need to decide if sockets based solution provides too much complexity to runc or not. There's probably also a way to try to achieve it with fifos but it would be hacky and I'm not sure how stable.

Keep in mind that this is not only a technical question but we would need a way to define what is going on when there is nothing attached to runc. For example if there is a logger setup we do not want to lose any logs between attaches. When there is no logger we don't want to buffer but simply discard unless there is someone attached. We also dont' want to just block or error on writes because user applications likely don't know how to handle that. This would mean something like "stream": {"stdout": 10240, "stderr": false} added to config.

As it goes pretty complex, I think we should consider leaving this out of runc. This can always be handled in an upper layer. If you want to crash the daemon you could write something intermediate that only handles the streams.

@LK4D4 @crosbymichael

[wking]

On Fri, Aug 14, 2015 at 10:58:58AM -0700, Tõnis Tiigi wrote:

As it goes pretty complex, I think we should consider leaving this
out of runc.

I agree that reattaching a higher-level monitor to streams associated
with a given container process sounds like a higher-level problem.

If reattaching to streams or getting nonblocking streams on monitor
crashes proves too difficult, it might be worth having a separate
process acting as a pub/sub hub for streams that cycles independently
from the monitor process. Then the new monitor would just have to
subscribe to the streams for running containers when it started up.

[mudverma]

@tiborvass Hi, I was just wondering about how Docker is integrating with runc? Is there some internal work on getting rid of parent child relationship b/w daemon-runc-containers?

runc daemon integration was targeted for 1.9? So there must be some work on that already.

[tiborvass]

@mudverma It didn't make it in 1.9, but @LK4D4 or @crosbymichael can say more about this.

As far as this issue is concerned, I think it makes sense to leave reattaching out of runc like @tonistiigi said. I'm closing this issue.