Slow process spawning using `Command::spawn` with older GLIBC due to use of `fork`

[Kobzol]

Hi! I have recently noticed that spawning new child processes using Command::spawn can be quite slow on older GLIBC versions on Linux. I have tracked this down to the usage of posix_spawn here.

Problem description

posix_spawn uses fork on older GLIBC versions (<2.24). While fork is supposed to be fast because it uses copy-on-write, it can actually be quite slow, because it still has to do nontrivial things, like setting up page tables. This gets problematic if the process has a lot of memory allocated, because then many virtual memory pages are populated and fork gets really slow.

I ran into this on an HPC cluster that uses GLIBC 2.17. I'm working on a distributed runtime that executes many (often short-running) tasks on the cluster, and from performance profiles it seemed like most of the time is actually spent inside fork, which looked really weird. Spawning processes if crucial for our runtime, so this poses a problem, because the spawning sometimes adds too much overhead.

Benchmark

I constructed a simple benchmark that allocates ~4 GiB memory (to simulate a process with populated page tables) and then spawns 100 processes that immediately exit (they execute sleep 0).

use std::process::Command;
use std::time::Instant;

fn main() {
    // Populate page tables
    let mut v: Vec<u32> = vec![];
    for i in 0..1024 * 1024 * 1024 {
        v.push(i * i + 1);
    }

    let start = Instant::now();

    for _ in 0..100 {
        Command::new("sleep").arg("0").output().unwrap();
    }

    println!("{} ms", start.elapsed().as_millis());
    println!("{}", v[1000] + v[11111]);
}

On my laptop with GLIBC 2.33, it takes under 100 ms to run the benchmark, while on the HPC cluster this can sometimes take up to 10 seconds! Which is quite a lot for spawning 100 processes. (Using spawn instead of output doesn't really change anything).

Potential solution

posix_spawn has a flag called POSIX_SPAWN_USEVFORK. When used, vfork will be used instead of fork on GLIBC versions before 2.24 (in newer versions it uses a different approach using clone, which seems to be fast). If I understand it correctly, vfork is designed for use cases when you fork and then immediately call exec*. It avoids page table manipulations and should run in constant time independent of the process memory usage.

There are some benchmarks that demonstrate that this flag can actually help with process spawning performance (https://github.com/rtomayko/posix-spawn#benchmarks). Go also tries to avoid fork from version 1.9, which had some nice positive effects. A similar request was also discussed for Python a few years ago (https://bugs.python.org/issue34663), where it was rejected.

I wasn't sure how to use a custom libstd for the benchmark above, so I wrote a very simple C++ program that tries to emulate the Rust benchmark above. First I used posix_spawn without POSIX_SPAWN_USEVFORK, it ran very fast locally with recent GLIBC, but quite slow on the cluster with prehistoric GLIBC. When I added the vfork flag to posix_spawn, suddenly process spawning became very fast even on the cluster. So it seems that this flag can actually solve the problem of slow process spawning on older systems.

C++ version of the benchmark (compiled with `gcc -O2`)

#include <vector>
#include <iostream>
#include <chrono>
#include <spawn.h>

#define CHECK(err) \
{                  \
    int ret = err; \
    if (ret != 0) {    \
        errno = ret; \
        std::cerr << ret << std::endl; \
        perror("fail"); \
        exit(1); \
    }\
}

int main() {
    std::vector<int> vec;
    for (int i = 0; i < 1024 * 1024 * 1024; i++) {
        vec.push_back(i * i + 1);
    }

    auto start = std::chrono::system_clock::now();

    for (int i = 0; i < 100; i++) {
        char* envp[] = {NULL};
        char* args[] = {
            "/usr/bin/sleep",
            "0",
            NULL
        };

        posix_spawnattr_t attr;
        CHECK(posix_spawnattr_init(&attr));
        CHECK(posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF | POSIX_SPAWN_SETSIGMASK/* | POSIX_SPAWN_USEVFORK */));

        pid_t pid;
        CHECK(posix_spawn(
            &pid,
            "/usr/bin/sleep",
            NULL,
            &attr,
            &args[0],
            &envp[0]
        ));
        CHECK(posix_spawnattr_destroy(&attr));
    }

    auto end = std::chrono::system_clock::now();
    auto diff = std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count();
    std::cout << diff << " ms" << std::endl;

    std::cout << vec[1000] + vec[10231] << std::endl;

    return 0;
}

Proposal

Would it be possible to use the POSIX_SPAWN_USEVFORK flag here to increase the performance of process spawning on older systems?

At first glance, it seems like a simple change, but I'm not a Linux wizard and I don't know if there are any issues with vfork or this flag. It seems that its use was somehow controversial in the past. Furthermore, the flag seems to be GNU specific, I'm not sure if that poses a problem.

[cuviper]

When Python was considering this flag, Florian Weimer (a glibc maintainer) said:

I wouldn't bother with POSIX_SPAWN_USEVFORK on GNU/Linux. Current versions of glibc always use a vfork-style clone call, so there would be a difference on older versions only. But there, the vfork code has subtle bugs, so using POSIX_SPAWN_USEVFORK there is not a good idea, either.

[Kobzol]

Well, I don't know enough about vfork, but I read many "opinions" about it online, both positive and negative. It seems to be on the level of urban legends sometimes, and it's unclear to me what is still relevant in "modern" kernel versions. After all, Go decided to use it. So I wanted to get more opinions here :)

But I understand if there's pushback against using it, even if just to stay on the safe side. After all it will be fixed on its own once the old glibc versions vanish. And I suppose that passing the flag optionally also isn't an option.

It would probably be enough for us to use a custom libstd with the flag enabled. As a semi-related question, what's the proper way of using a custom libstd build? Link to a modified libstd.so library with something like LD_PRELOAD? Or build a whole custom toolchain including rustc and use it with e.g. rustup link?

(If it's too off-topic and you don't see any hope for POSIX_SPAWN_USEVFORK . feel free to close this issue).

[cuviper]

My interpretation is that glibc itself was buggy in its vfork, not the kernel syscalls, or else they would be talking about the relevant kernel versions as well. Go bypasses any libc and instead uses syscalls directly.

If you can't get a newer glibc sysroot on your cluster, maybe you could use Rust targeting linux-musl?

It would probably be enough for us to use a custom libstd with the flag enabled. As a semi-related question, what's the proper way of using a custom libstd build? Link to a modified libstd.so library with something like LD_PRELOAD? Or build a whole custom toolchain including rustc and use it with e.g. rustup link?

You'd need a whole toolchain build, because Rust libraries are only ABI compatible with the exact same compiler.

When Cargo's build-std feature is stable, you could probably use that after patching the std source.

[Kobzol]

My interpretation is that glibc itself was buggy in its vfork, not the kernel syscalls, or else they would be talking about the relevant kernel versions as well. Go bypasses any libc and instead uses syscalls directly.

Oh, that actually explains a lot, I didn't realize that. And since my problem is that my glibc is too old (and probably buggy)... Well, tough luck.

If you can't get a newer glibc sysroot on your cluster, maybe you could use Rust targeting linux-musl?

Good idea. Although we use jemalloc and I wasn't able to get it working with musl. But it's an option, it seems that musl does The Right Thing™.

You'd need a whole toolchain build, because Rust libraries are only ABI compatible with the exact same compiler.

Thanks, good to know that :)

Well, I guess that since this issue is quite esoteric and it probably doesn't warrant a change in the standard library, I'm going to use your hints to build a custom rustc toolchain and close this issue. Maybe it will be useful to someone else in the future.

[cuviper]

When Python was considering this flag, Florian Weimer (a glibc maintainer) said:

I wouldn't bother with POSIX_SPAWN_USEVFORK on GNU/Linux. Current versions of glibc always use a vfork-style clone call, so there would be a difference on older versions only. But there, the vfork code has subtle bugs, so using POSIX_SPAWN_USEVFORK there is not a good idea, either.

[Kobzol]

Well, I don't know enough about vfork, but I read many "opinions" about it online, both positive and negative. It seems to be on the level of urban legends sometimes, and it's unclear to me what is still relevant in "modern" kernel versions. After all, Go decided to use it. So I wanted to get more opinions here :)

But I understand if there's pushback against using it, even if just to stay on the safe side. After all it will be fixed on its own once the old glibc versions vanish. And I suppose that passing the flag optionally also isn't an option.

It would probably be enough for us to use a custom libstd with the flag enabled. As a semi-related question, what's the proper way of using a custom libstd build? Link to a modified libstd.so library with something like LD_PRELOAD? Or build a whole custom toolchain including rustc and use it with e.g. rustup link?

(If it's too off-topic and you don't see any hope for POSIX_SPAWN_USEVFORK . feel free to close this issue).

[cuviper]

My interpretation is that glibc itself was buggy in its vfork, not the kernel syscalls, or else they would be talking about the relevant kernel versions as well. Go bypasses any libc and instead uses syscalls directly.

If you can't get a newer glibc sysroot on your cluster, maybe you could use Rust targeting linux-musl?

It would probably be enough for us to use a custom libstd with the flag enabled. As a semi-related question, what's the proper way of using a custom libstd build? Link to a modified libstd.so library with something like LD_PRELOAD? Or build a whole custom toolchain including rustc and use it with e.g. rustup link?

You'd need a whole toolchain build, because Rust libraries are only ABI compatible with the exact same compiler.

When Cargo's build-std feature is stable, you could probably use that after patching the std source.

[Kobzol]

My interpretation is that glibc itself was buggy in its vfork, not the kernel syscalls, or else they would be talking about the relevant kernel versions as well. Go bypasses any libc and instead uses syscalls directly.

Oh, that actually explains a lot, I didn't realize that. And since my problem is that my glibc is too old (and probably buggy)... Well, tough luck.

If you can't get a newer glibc sysroot on your cluster, maybe you could use Rust targeting linux-musl?

Good idea. Although we use jemalloc and I wasn't able to get it working with musl. But it's an option, it seems that musl does The Right Thing™.

You'd need a whole toolchain build, because Rust libraries are only ABI compatible with the exact same compiler.

Thanks, good to know that :)

Well, I guess that since this issue is quite esoteric and it probably doesn't warrant a change in the standard library, I'm going to use your hints to build a custom rustc toolchain and close this issue. Maybe it will be useful to someone else in the future.