Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tini Exits Too Early Leading to Graceful Termination Failure #180

Open
jschwinger233 opened this issue Feb 27, 2021 · 5 comments
Open

tini Exits Too Early Leading to Graceful Termination Failure #180

jschwinger233 opened this issue Feb 27, 2021 · 5 comments

Comments

@jschwinger233
Copy link

jschwinger233 commented Feb 27, 2021

TL;DR

When dealing with multi sub processes, tini is likely to return without waiting for all children to exit, and that drives kernel send SIGKILL to the remaining processes under the same pid namespace, going against the design principle for tini: allowing for signal forwarding and graceful termination.

Steps to Reproduce

  1. get a tini binary

  2. prepare a python script as followed:

# ppp.py
import os
import time
import signal
pid = os.fork()
if pid == 0:
    signal.signal(15, lambda _, __: time.sleep(1))
    cpid = os.fork()
time.sleep(1000)

Note that I use time.sleep(1) to act as if there is some time consuming operation for graceful termination.

  1. make sure the script ppp.py and tini bin tini are in the $(pwd), then run a docker container as followed:
docker run -d --rm -v $(pwd):/src -w /src python /src/tini -g -s -- python /src/ppp.py
  1. inspect process tree
root     335889      1  0 Feb24 ?        00:05:21 /usr/bin/containerd
root       2707 335889  0 23:02 ?        00:00:00  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/7ea94ebea673367a6e31f136115dcbe5d0d18bc4c343ca5e834549f30ba7b189 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       2747   2707  0 23:02 ?        00:00:00      \_ /src/tini -g -s -- python /src/ppp.py
root       2810   2747  3 23:02 ?        00:00:00          \_ python /src/ppp.py
root       2840   2810  0 23:02 ?        00:00:00              \_ python /src/ppp.py
root       2841   2840  0 23:02 ?        00:00:00                  \_ python /src/ppp.py
  1. use strace(1) to watch the last python process 2841: strace -fTp 2841, then send SIGTERM to tini: kill -15 2747

What We Expected

Since there is no multiple process group, and we specified the tini with -s -g, the actions for tini should have been:

  1. forward SGTERM to all the python processes
  2. wait for all python processes to exit (graceful termination)
  3. exit as the last the process in the pid namespace

What We Got

However, we could simply observed that the second (2840) and the third (2841) python processes received SIGKILL as soon as SIGTERM came, demonstrating that tini didn't wait for their exits so that their graceful termination failed.

Root Cause and Suggestions

  1. Who sent the SIGKILL?
    Kernel

If the "init" process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal.

  1. Why tini didn't wait?

    tini/src/tini.c

    Lines 546 to 560 in b9f42a0

    current_pid = waitpid(-1, &current_status, WNOHANG);
    switch (current_pid) {
    case -1:
    if (errno == ECHILD) {
    PRINT_TRACE("No child to wait");
    break;
    }
    PRINT_FATAL("Error while waiting for pids: '%s'", strerror(errno));
    return 1;
    case 0:
    PRINT_TRACE("No child to reap");
    break;

    Look at the second branch case 0, whose semantic in the waitpid(2) is:

waitpid(): on success, returns the process ID of the child whose state has changed; if WNOHANG was specified and one or more child(ren) specified by pid exist, but have not yet changed state, then 0 is returned.

So waitpid(-1, NOHANG)=0 means there is no "waitable" child(ren), but there IS child(ren), and that's when tini exits: with children still alive

  1. Is this expected?
    You might argue that, as long as the direct child process has decent behavior of handling SIGTERM, such as the first python 2810 should wait for the second python 2840 before quitting, the tini is flawless.
    Well that's quite true, but provided that the direct child is capable of grappling with graceful termination and so on, what's the point of installing tini as pid 1 in the container? In that case, we should run the application process as pid 1, without tini.

  2. Improvement Suggestions
    There are many ways to tackle the issue, and the principle is as simple as NOT exit until all children are gone.
    The syscall waitpid(2) already offers us ability to distinguish between "there is no child" and "there is no waitable children", so we just follow the doc, and change the tini exit condition.

@jschwinger233
Copy link
Author

jschwinger233 commented Feb 27, 2021

My friend @Zheaoli is working on a PR to address the issue.

@jschwinger233 jschwinger233 changed the title Wrong Usage in waitpid(2) Leads to Graceful Termination Failure tini Exit Too Early Leading to Graceful Termination Failure Feb 27, 2021
@jschwinger233 jschwinger233 changed the title tini Exit Too Early Leading to Graceful Termination Failure tini Exits Too Early Leading to Graceful Termination Failure Feb 27, 2021
@vsxen
Copy link

vsxen commented Mar 7, 2021

对我来说更可怕的是容器停不掉
ex:
https://www.likakuli.com/posts/docker-pod-terminating/

@Hello-Linux
Copy link

Hello-Linux commented Jul 17, 2022

@jschwinger233 @vsxen @antonmos @zimbatm When will this problem be fixed? If tini exits early without waiting for the child process, then the function of causing tini to forward signals is meaningless

@yosifkit
Copy link

tini is only designed to wait for its direct child to exit; if that child spawned its own children, then it should be signaling/waiting for them before exiting. tini will reap zombies, but not wait explicitly for the nested children to exit (since it only knows of their existence once they become a zombie, i.e. they exited and their parent exited without waiting on them).

@kiddingl
Copy link

kiddingl commented Apr 23, 2024

我也遇见了这个问题,tini执行了个bash脚本,kill tini进程后会立即停止bash脚本,内部产生的子进程无法收到信号,导致tini快速退出,子进程被强制终止
后来内部命令使用了exec执行,确实可以解决这个问题,但是exec 命令后面使用管道(exec sleep 200 | ls)这种就又无法生效了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants