Handling of zombie processes would be useful

[tomwys]

This image does not clean up zombie processes. Zombie processes may appear when job fails.

[loa]

Friendly paste :)

Phusion - Docker and the PID 1 zombie reaping problem

[tobowers]

I'm seeing some zombie ssh processes as well.

[ghost]

zombies accumulate indeed

$ ps fauwwwx | grep defunct | wc -l
68

[krallin]

Hi there,

I ran into this exact issue and created https://github.com/krallin/tini as a solution. The idea was to have a drop-in "zombie reaper" that doesn't do anything else (in comparison phusion's my_init does quite a lot).

My Dockerfile looks like this:

FROM jenkins

USER root
ENV TINI_VERSION v0.3.4
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini

USER jenkins
ENTRYPOINT ["/tini", "--", "/usr/local/bin/jenkins.sh"]    

---

@ndeloof - Would it make sense to make a PR to add tini to the image?

Obviously, pulling the binary directly (which is what I pasted above) may not be desirable, but I have code here that can be used to build tini in the Dockerfile and cleanup the environment. I'm also planning on building debs for it.

https://github.com/krallin/tini-images/tree/master/autogen

[lpellegr]

I had the same issue and it was making some Jenkins jobs failing because the job was involving some tests that were checking running processes. I applied the solution proposed by krallin and it is now working. It would be great to include a solution in the base image.

[ndeloof]

I wonder why this appear as a docker image issue, not a generic jenkins issue. Jenkins is (supposed to) kill all processes after build completion, so should not generate such zombie processes (until there's something docker-specific I'm missing)

proposed tini workaround just seems to hide the root issue

[tomwys]

@ndeloof This is not correct. Lets consider following process tree: jenkins -- bash -- python. If bash dies unexpectedly (ie. kill -9) python process will have no parent. In such case parent will be changed automatically by kernel to PID 1 (init process in normal Linux based system). In such case Jenkins can and should clean up after bash. But python is not Jenkins business anymore, because it is not in his process tree anymore. So python won't be cleaned by Jenkins, but it will be detected by init process as new child and cleaned up by init.

But in your container PID 1 is Jenkins. So this python process will be stuck as zombie process forever.

[ndeloof]

Jenkins do rely on inherited env to detect child processes created by a job and kill them on build completion. Need to experiment with this and try to find a jenkins-centric fix for it, better than hacking the docker image. Thanks for the detailed description

[krallin]

@ndeloof ,

Note that zombie reaping wouldn't necessarily be that useful in Jenkins outside of a Docker environment, considering zombies are usually reaped by init (though on newer kernels you could register Jenkins as a subreaper).

Cheers,

[ndeloof]

tini indeed seems to be the sole option we have. Sorry I didn't understood the issue first time I commented here
@krallin how can a process be registered as a subreaper ? IIUC this require a system call from the process to become subreaper, there's no CLI tool for this purpose, right ?

[krallin]

@ndeloof

Yes, it requires a system call. It's pretty easy to write a CLI tool to do it (it persists across a fork). You'd also need to actually reap zombies though.

Cheers,

[gohonsen]

@krallin
I use tini start jenkins, but it still became zombie process after docker stop jenkins container:

[root@docker32-111 ~]# docker ps -a
231890d0b055 jenkins:2.7.1 "/bin/tini -- /usr/lo" 4 weeks ago Exited (4294967295) 1 hour ago big_rosalind

[root@docker32-111 ~]# ps -elf | grep -v grep | grep defunct
0 Z root 15524 15507 6 80 0 - 0 exit 10月13 ? 1-04:40:10 [java]

pid 15507 is tini process:
[root@docker32-111 ~]# ps -elf | grep -v grep | grep 15507
4 S root 15507 1 0 80 0 - 0 wait 9月29 ? 00:01:38 [tini]
0 Z root 15524 15507 6 80 0 - 0 exit 10月13 ? 1-04:40:10 [java]

and the tini cmd name displays wrong, the normal is "/bin/tini -- /usr/local/bin/jenkins.sh" but not "[tini]"

there are lots of child processes of 15524, and the status all is D
[root@docker32-111 ~]# ps -elf | grep 15524
1 D root 522 15524 0 80 0 - 11527716 sleep_ 10月31 ? 00:00:00 java -jar jenkins.war
1 D root 839 15524 0 80 0 - 11527716 sleep_ 10月31 ? 00:00:00 java -jar jenkins.war
......

here is Dockerfile entrypoint:
ENTRYPOINT ["/bin/tini", "--", "/usr/local/bin/jenkins.sh"]

Is there any good suggestions ?