How to spawn and clean up subprocesses correctly?

[Atry]

Hi all, recently I am working on HSL Subprocess API. However, there are some issues in existing implementations of PHP style subprocess API. The rest of this article includes some ideas based on private chat between @fredemmott and me.

Existing Implementation

Unlike PHP's single threading mode, HHVM's server mode runs in a multithreaded environment. Unfortunately, the fork system call is unsafe in multithreaded environment. As a workaround, HHVM introduced a setting Server.LightProcessCount. When LightProcessCount is greater than zero, HHVM would pre-fork some worker processes before additional threads creation, and proc_open or other functions would send a request to a worker process via a UNIX socket, letting the worker process to actually spawn the subprocesses that proc_open asked.

Issues

Issue 1: Waitpid from the main process

As demonstrated in #9015 (comment) , worker processes are creating zombies, because the current pcntl_waitpid implementation directly calls wait4 given a PID created by the worker processes, therefore, the PID is actually the grand child of the HHVM's main process, while wait4 requires the PID is a direct child of the caller process.

Issue 2: Waitpid from the wrong worker process

Alternatively, we can also let worker processes, not the main process, call wait4. However, currently the worker processes are associated with HHVM threads, not the PID. That is to say, even if we let a worker process execute wait4, if proc_open is called in one thread then pcntl_waitpid is called by another thread, then it is very likely to turn the subprocess into a zombie because wait4 might not be called by the same worker process.

Solutions

Option 1: Switch to the native posix_spawn

This could eliminate some but not all the motivation behind the pre-forked light processes, because unlike fork, posix_spawn is expected to be compatible with multithreading mode. However the other motivation of pre-forked light processes is to separate the permissions, as when running HHVM in CLI server mode, the lightprocess master is part of the CLI client process, not the HHVM server.

Since anyway we must spawn subprocesses from processes other than the CLI server process, either from the CLI client process or light weight worker processes, we would still have to deal the issue to find the right caller of wait4, even if we switch the underlying implementation to the native posix_spawn.

Option 2: Use exact a single worker process

This approach would result bad performance because currently some of worker processes are used to turn blocking operations into non-blocking, including OS\open() and anything using the HSL_CLI_INVOKE.

Option 3: Build an additional pid -> worker process mapping

If we build an additional pid to worker process mapping, we could ensure a waitpid command is always handled by the same worker process that created the PID. However, this approaches seems conflicting with existing usage of LightProcess::setThreadLocalAfdtOverride, which assigns a light process to a thread-local variable.

Option 4: Mimic one process per request behavior

The current thread to worker process mapping works well if the waitpid and proc_open calls are from the same request (and we are going to move the wait4 call to the associated worker process), because currently we always associate the same thread with a request. For subprocesses that live longer than a request, we might want to turn the subprocess into a daemon process, mimicking PHP's one process per request behavior, so that the PID would always be cleaned up.

Related topics

• How can we wait for the exit code of a subprocess asynchronously?
• light process - source file
• High level HSL subprocess API proposal
• Expand OS\ design notes
• Low level subprocess API design

[fredemmott]

Going to split into several subthreads:

Issue 2: Waitpid from the wrong worker process

Except for pcntl_waitpid() which is buggy, this is fine as long as the waitpid() is from the same request as the spawn. I strongly believe that's a desirable requirement, but that is a separate discussion.

All lightprocess requests for a request currently come from the same thread; if that changes in the future, we do support a distinct concept of 'request local', and we would need to use that instead.

[fredemmott]

that is a separate discussion.

Issue 4 comment:

#9016 (comment)

[fredemmott]

Switch to the native posix_spawn

This is probably long-term desirable; however:

• we currently support platforms where this doesn't exist
• it is unclear if it adequately handles forking multithreaded programs on all the platforms we support where it does exist; my understanding of https://bugs.chromium.org/p/chromium/issues/detail?id=179923 is that there may be practical problems here

[fredemmott]

Issue 1: Waitpid from the main process

This is a plain bug. Lightprocess::pcntl_waitpid never actually uses lightprocesses, so will only work correctly in standalone CLI mode. Other functions, e.g. waitpid do work correctly

[fredemmott]

Option 2: Use exact a single worker process
Option 3: Build an additional pid -> worker process mapping

This approach would result bad performance because currently some of worker processes are used to turn blocking operations into non-blocking

Yep, all the CLI server stuff. This is a pre-existing problem, but we won't be making it worse here.

Naively, I would want a single request to be able to use multiple lightprocess workers simultaneously; I don't know how practical this is when looking at all requests together though - e.g:

• could it allow a single request to starve all the workers?
• is increasing the number of lightprocess workers a managable solution from the perspective of resource usage on the webservers?

[fredemmott]

Option 4: Mimic one process per request behavior

I strongly believe this is the desirable approach:

• it would be consistent with other functions like setlocale(), chdir()
• in general, this is the PHP execution model that we inherited, as PHP traditionally literally is a process per request
• it retains request isolation
• it keeps behavior consistent between CLI server and standalone CLI mode

Doing otherwise would also require:

• ensuring that the waitpid is executed with the permissions of the request doing the waitpid, not the request that did the spawn (especially in CLI server mode)
• addressing inter-request FD sharing, or consciously deciding it's unneeded; doing this would have similar security questions to investigate as waitpid

I also don't think this should be considered a blocker on implementation:

• we're not aware of use cases for cross-request spawn+wait. It's also worth noting that in multi-server environments, this would only be usable if we guarantee that the request with the waitpid goes to the same server that did the spawn
• without use cases, we are unlikely to design it well
• it doesn't seem likely that supporting it now will reduce work compared to doing it later

[fredemmott]

To turn the subprocess into a daemon process

IMO this should either be up to the process itself, or an explicit spawn-time option; the default should probably to terminate the process at the end of the request

[fredemmott]

To expand a little: I don't think that requests have enough control over their lifetime - or how far into it the spawn is done - for "wait on it if the request is still running, run until end in the background if not" to be a useful behavior. IMO every request should either be spawned with the intention of waiting for it in the current process (in which case it should be killed+waited on request end if needed), or a 'fire and forget' model (in which case it should be detached and reparented to init when spawned)

[Atry]

In Linux, automatically terminating a child process when its parent process exit is usually achieved by SIGHUP, which requires the parent process to be the session leader who would close the controlling terminal on exit. However, this behavior is difficult to mimic in LightProcess, because we cannot create controlling terminal per request.

[fredemmott]

IIRC that is possible via the setsid/setpgid stuff, but it's also unneeded, as the HHVM master and lightprocess worker processes continue after the request has finished

[Atry]

Yes. We cannot really mimic every behavior like the one process per request mode.