Option to force use of posix_spawn

[adamgundry]

In a client project, we have a server that for performance reasons must not call fork under any circumstances (it has a very large resident set). Unfortunately since the removal of vfork, this has been hard to arrange (cf. #303), because the conditions under which posix_spawn works are opaque and if any of them fail, the code automatically falls back on fork. In our case it would be preferable for the whole createProcess call to fail.

Could process provide an option to disable the fork-exec path? Specifying this at compile time or even configure time is fine; I'm not sure if there is an easy way to override configure to treat fork as not available?

[bgamari]

I do wonder why fork is so problematic performance-wise. Afterall, modern fork implementations should be implemented via copy-on-write, meaning that the only residency incurred by the operation should be the duplicate page tables needed to represent the "copied" mappings. These mappings will ultimately be freed by exec. This is perhaps a bit inefficient, but I'd be surprised if it were prohibitive.

[adamgundry]

It is of course possible I've misunderstood something. But here is what I think is happening:

• The server has a very large residency (80-100GiB), as it is an in-memory database. It is handling requests concurrently, of which a small proportion (say 1%) require forking a child process. In the usual style, the server allocates frequently while processing requests. Most of the allocations are short-lived and do not escape the (large) nursery; major GCs are rare and expensive.
• With the change from vfork to fork we see a dramatic increase in CPU load and request latencies, but not much impact on residency.
• Under vfork semantics, each call will pause the parent process until the exec (which might manifest as a slight increase in latency for other requests, but in practice the time taken is negligible).
• Under fork semantics, each call will duplicate the page tables and mark them as read-only, then the parent can continue executing concurrently with the child, but memory writes in either process will trigger page faults so the COW pages can be unshared. Copying the page tables presumably incurs a global pause, but I've not found data on how long this is. But I think that in between fork and exec there are other requests being served concurrently by other threads, which will try to write to lots of other pages, incurring many page faults.

Does that sound plausible?

[bgamari]

With the change from vfork to fork we see a dramatic increase in CPU load and request latencies, but not much impact on residency.

Ahh, yes. Somehow I was under the impression that the problem was one of residency. A significant increase in CPU load is indeed plausible for precisely the reason that you describe.

I'm a bit torn on how best to handle this. A global flag for this sort of behavior seems wrong; however, adding a flag to CreateProcess seems quite inconvenient for your use-case.

I believe that setting ac_cv_func_fork=no during the configure step should be sufficient to avoid use of fork. This can be accomplished via either the environment or by the following in cabal.project:

package process
    configure-options: ac_cv_func_fork=no

[adamgundry]

Alas, setting ac_cv_func_fork=no results in a process package that builds but results in linker errors when trying to use it, because runProcess.c doesn't define any symbols in that case:

[6 of 6] Linking TestCreateProcess
./System/Process/Posix.o(.text+0x3c8c): error: undefined reference to 'runInteractiveProcess'
./System/Process.o(.text+0x1673): error: undefined reference to 'terminateProcess'
./System/Process.o(.text+0x22e7): error: undefined reference to 'getProcessExitCode'
./System/Process.o(.text+0x3672): error: undefined reference to 'waitForProcess'
collect2: error: ld returned 1 exit status
ghc-9.10.1: `gcc' failed in phase `Linker'. (Exit code: 1)

That said, this should be easy enough to fix.