Tracking Child Processes (#2568)

[fbriol]

Modification allowing to take into account the child processes created from the workers.

[mrocklin]

Hrm, it looks like it takes a few milliseconds in order to compute this. This may be too expensive to have on by default. We run this fairly frequently.

In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: %timeit proc.children()
4.87 ms ± 161 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[fbriol]

The time-consuming step is the reading of the process table. First line of the psutil.Process.children function.

In [1]: import psutil

In [2]: proc = psutil.Process()

In [3]: %timeit proc.children()
8.79 ms ± 36.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [4]: %timeit psutil._psplatform.ppid_map()
8.59 ms ± 101 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

One of the possible solutions, maybe if you have a better idea, is to delegate the calculation to a thread. For example,

import threading
import psutil
import time


class TrackChildren(threading.Thread):
    def __init__(self, proc, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self._proc = proc
        self._cpu = 0
        self._mem = 0
        self._children = set()
        self._lock = threading.Lock()
        self._event = threading.Event()

    def stop(self):
        self._event.set()

    def values(self):
        with self._lock:
            return self._cpu, self._mem

    def run(self):
        while not self._event.is_set():
            cpu = 0
            mem = 0
            children = set(self._proc.children(True))
            # "self.children" tracks the subprocesses to calculate the CPU
            # usage correctly. Otherwise, the computed CPU usage would always
            # be 0 as no time interval would exist for the calculation
            # (cf. psutil.Process.cpu_percent).
            if children:
                new_children = children - self._children
                if new_children:
                    self._children.update(new_children)
                for child in list(self._children):
                    # The inspected process may die during its introspection.
                    try:
                        with child.oneshot():
                            cpu += child.cpu_percent()
                            mem += child.memory_info().rss
                    except (psutil.NoSuchProcess, psutil.ZombieProcess):
                        self._children.discard(child)
            with self._lock:
                self._cpu, self._mem = cpu, mem
            time.sleep(0.1)

In this case, the reading of the numbers is much faster.

t = TrackChildren(psutil.Process())
t.start()
%timeit t.values()
298 ns ± 4.68 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)