Zombie subprocesses

[asantos00]

Currently, when creating a subprocess using Deno.run, the process stays in "zombie mode" even after it is killed.

I've created a gist that reproduces this bug, and talked with @lucacasonato that confirmed it.

https://gist.github.com/asantos00/bdd4357cb91f18b368085eb44105ea97

[caspervonb]

Example from the gist

const command = ['deno', 'run', './script.ts'];

const p1 = Deno.run({ cmd: command })
const p2 = Deno.run({ cmd: command })
const p3 = Deno.run({ cmd: command })
const p4 = Deno.run({ cmd: command })

p1.kill(Deno.Signal.SIGKILL)
p2.kill(Deno.Signal.SIGKILL)
p3.kill(Deno.Signal.SIGKILL)
p4.kill(Deno.Signal.SIGKILL)

setInterval(() => {
  // just to keep it running so you can `ps aux | grep deno` to check the zombie processes
}, 1000);

[bnoordhuis]

For background: UNIX expects you to reap zombies with waitpid(2) when you receive a SIGCHLD signal.

This would normally be an easy fix - set the SIGCHLD signal handler to SIG_IGN and let the kernel handle it - except....

tokio::process::Child installs its own SIGCHLD handler that records events into a queue. There is nothing polling the command objects and so events stay in the queue and zombies aren't reaped.

A crude hack that papers over the issue:

diff --git a/cli/ops/process.rs b/cli/ops/process.rs
index 60a6d5095..b259d9531 100644
--- a/cli/ops/process.rs
+++ b/cli/ops/process.rs
@@ -114,6 +114,13 @@ fn op_run(
 
   // Spawn the command.
   let mut child = c.spawn()?;
+
+  #[cfg(unix)]
+  unsafe { libc::signal(libc::SIGCHLD, libc::SIG_IGN); } // Reset signal handler.
+
   let pid = child.id();
 
   let stdin_rid = match child.stdin.take() {

It's not a good fix however because of the race window between the spawn and signal calls.

So far I haven't been able to come up with a better idea than poll child objects in the resource table at a fixed interval.

[Soremwar]

I can reproduce this in latest canary. Linux reports all the child processes as zombies

[nktpro]

This is definitely still an issue in latest deno release 1.7.0 and it's a big problem for apps that constantly spawn a lot of child processes. We did end up with non-responsive servers after a huge amount of zombie processes were leaked.

Is there any workaround we can do temporarily for now at the application layer before it's fixed in deno runtime, other than letting the deno parent process dies when any child process is killed (which is of course not ideal for long running servers)?

[muzuiget]

This issue still happens on v1.7.1

[satyarohith]

This still seems to be an issue on 1.9.2.

[ebebbington]

Happens when i run firefox as a subprocess on windows

[ebebbington]

Any hope on getting this addressed?

[GJZwiers]

As an alternative, this issue does not appear to happen when using the Deno.Command API.

[Hajime-san]

We should migrate to Deno.Command API because of will be soft-removed in Deno 2.0.
https://deno.land/api@v1.43.6?s=Deno.run