Very slow process fork if RLIMIT_NOFILE is too high

[dlangBugzillaToGithub]

trikkuz reported this on 2024-04-27T09:30:04Z

Transfered from https://issues.dlang.org/show_bug.cgi?id=24524

Description

In `std.process`, the `spawnProcessPosix` function, used by all functions that start a process, has a flag `Config.Flags.inheritFDs`. If set to false, it reads the maximum limit of files that can be opened by a process, `RFILE_NOFILE`, and then creates a structure for each one and finally iterates through them.

Normally, the `RFILE_NOFILE` limit is set to 1024 or anyway a relatively low number. The hard limit depends on the system; in some, it is 2^20, in others even 2^30.

When starting a container with Docker, the limit is raised to the maximum allowed, and this inevitably slows down the process visibly (because it has to allocate 2^20 (or 2^30) elements and then iterate through them. An user reported that on a system with 2^30 as the hard limit, it becomes literally unusable.

There would be a faster way to do this by reading the open file descriptors from `/dev/fd` with the code below, but we are inside a `@nogc` block, so another solution must be found.

`
import std.file : exists, isDir;

if (exists("/dev/fd") && isDir("/dev/fd"))
{
    import std.file : dirEntries, SpanMode;
    import std.path : baseName;
    import std.algorithm : map, filter;
    import std.array : array;
    import std.string : indexOfNeither;
    import std.conv : to;

    // List all file descriptors except stdin, stdout, stderr, and the pipe
    auto fds = dirEntries("/dev/fd", SpanMode.shallow)
        .map!(d => d.name.baseName)
        .filter!(d => d.indexOfNeither("0123456789") < 0)
        .map!(d => d.to!int)
        .filter!(d => d >=3 && d != forkPipeOut)
        .array; // Copy to avoid closing while iterating

    // Close only the file descriptors that are open.
    foreach(fd; fds)
        close(fd);
}
`

[dlangBugzillaToGithub]

dlang-bot commented on 2024-04-27T22:57:25Z

@trikko updated dlang/phobos pull request #8990 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" fixing this issue:

- Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high

https://github.com/dlang/phobos/pull/8990

[thewilsonator]

Was fixed by #9077