[Question] How to generate zombie processes to test PID 1 problem.

[EMChamp]

I would like to gain some insight into the docker zombie process. Is there a simple way to produce a zombie in a docker container to see if it is reaped or not?

I am trying to create a zombie process that will persist even after the container is closed (that's what I understand the problem is in any case) so I can prove that using dumb-init as my ENTRYPOINT will fix the problem. However I haven't been able to show that the zombie process I created will stay no matter how I terminate my container or no matter how i terminate the parent of the zombie process.

I'm running nvidia/cuda:7.5-devel for the base image and docker v1.12.3. I've created a zombie in my container by having this script run as PID 1 in my dockerfile via CMD:

#!/bin/bash
# Spawn a zombie process whose parent is this script which is PID 1 in a container.
(sleep 1 & exec /bin/sleep 200000) &

tail -f /dev/null 

I started my test by launching 2 containers, 1 with dumb init as PID 1 (which launches my script) and 1 with the script as PID 1.

I can see the zombies here

:~/dockerfiles$ ps -ef | grep sleep
root     30599 30597  0 16:24 ?        00:00:00 /bin/sleep 200000
root     30600 30599  0 16:24 ?        00:00:00 [sleep] <defunct>
root     30746 30722  0 16:24 ?        00:00:00 /bin/sleep 200000
root     30747 30746  0 16:24 ?        00:00:00 [sleep] <defunct>

but when I try to kill the parents either outside of inside of the container, the zombies are reaped even in the container not running dumb-init.

I'm looking for a way to reproduce this PID1 problem so I can show that dumb-init is doing this job because so far I haven't been able to show that zombies will stick to the host OS after the containers are killed, on the contrary the zombie processes are immediately reaped when I kill their parent process even if the container is still running.

[samrocketman]

You should start a container with sshd as PID1. SSH to the container and test your script. If PID1 dies then a container should clean itself up so you won't see the issue that way. Think of a long running daemon in a container that rugularly spawns children (which also spawn children of children). The PID1 issue can affect long running daemons with infinite resource growth (or to a defined limit for the container).

[chriskuehl]

Thanks for the question!

As far as I know, you're correct that zombie processes won't ever prevent a container from being stopped (and won't somehow stick to the host).

The important things to understand about zombie processes (at least as I understand them) are:

• Every process becomes a zombie when it exits, if only momentarily. They get "reaped" when the process's parent calls wait on them. The purpose of this is so that the parent can collect the child's exit status. In well-behaved systems, children which exit are usually reaped immediately by their parent.
• If a process's parent exits at some point, the child is re-parented under PID 1 (the "init" system). There's an expectation that the init system will wait on any zombie processes that become its children, since generally lots of processes end up parented under PID 1. This expectation is often violated in Docker (since random scripts/apps are often PID 1), and so Docker containers tend to build up lots of zombies.
• Zombies aren't particularly dangerous. They're not still running, and they don't take CPU time or memory (beyond the entry in the process table), and I don't know of any real dangers except running out of PIDs.
• dumb-init (and other init systems) can't help with all zombies, only orphaned zombies. If your app spawns a lot of direct children and never waits on them, they'll be zombies, but not parented under PID 1, so you won't get any help from your init system to clean them up.

I'd always suggest to use a proper init system that reaps zombies, but I don't really see that as the main reason to use dumb-init. Rather, I think the signal behavior (which can actually prevent containers from responding appropriately to Ctrl-C and other attempts to stop them) is more important. This is talked about in some depth in the blog post about dumb-init (especially under "Trouble signaling PID 1").

To answer your questions directly,

Is there a simple way to produce a zombie in a docker container to see if it is reaped or not?

You need to create orphaned zombie processes, which means a process needs to spawn a child and then exit without reaping it. Here's an example Python script and Dockerfile:

FROM debian:stretch
RUN apt-get update && apt-get -y install dumb-init procps python3
COPY test.sh /root/
CMD ["/root/test.sh"]
#CMD ["dumb-init", "/root/test.sh"]

#!/usr/bin/env python3
import os
import subprocess


pid = os.fork()
if pid == 0:  # child
    pid2 = os.fork()
    if pid2 != 0:  # parent
        print('The zombie pid will be: {}'.format(pid2))
else:  # parent
    os.waitpid(pid, 0)
    subprocess.check_call(('ps', 'xawuf'))

If you're not using dumb-init, you'll see:

ckuehl@neon:/tmp/tmp.dvWXrDhmyZ$ docker run ckuehl-test                                                  
The zombie pid will be: 8
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  4.0  0.2  26804  9684 ?        Ss   08:13   0:00 python3 /root/test.sh
root         8  0.0  0.0      0     0 ?        Z    08:13   0:00 [python3] <defunct>
root         9  0.0  0.0  34504  2764 ?        R    08:13   0:00 ps xawuf

If you're using dumb-init, you'll see:

ckuehl@neon:/tmp/tmp.dvWXrDhmyZ$ docker run ckuehl-test       
The zombie pid will be: 8
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   4048   692 ?        Ss   08:13   0:00 dumb-init /root/test.sh
root         6  4.0  0.2  26804  9772 ?        Ss   08:13   0:00 python3 /root/test.sh
root         9  0.0  0.0  34504  2776 ?        R    08:13   0:00  \_ ps xawuf

so far I haven't been able to show that zombies will stick to the host OS after the containers are killed

To my knowledge, this isn't something that can happen once the container exits.

[samrocketman]

I defer to @chriskuehl answer as I am likely wrong. He knows more about the issue than I.

[EMChamp]

@chriskuehl Thanks for the detailed response! I was able to get a better picture of the problem through testing out your example. I guess I was creating zombie processes but not successfully orphaning them with my script which is why I was not seeing them after. Thanks again for the help.

[Stratus3D]

Is there a way of generating zombie processes using pure Bash? I'd like to test this on my container but I don't have Python installed. I've tried several things but I'm not sure the processes I've spawned are in the right state.