pageserver: spawning walredo process is slow

[jcsp]

Problem

On some pageservers we see >1s times to spawn the process.

Investigation Results

• Customer investigation https://neondb.slack.com/archives/C033RQ5SPDH/p1706787518630459?thread_ts=1706774416.482029&cid=C033RQ5SPDH

• walredo spawn latency is bimodal on most pageservers: some spawns are fast, taking tens of milliseconds, others asre slow, taking multiple seconds

• even though rust stdlib uses the efficient posix_spawn by default, we don't use it on pageservers because we use pre_exec() in close_fds()

DoD

• walredo process spawning latency is predictable
• acquisition of a walredo process for page reconstruction is < XXX milliseconds

Plan

Explore whether we can us posix_spawn; if so, ship to staging and observe whether it is a sufficient improvement. We can move the close_fds work into walredo startup, where we still trust the process.

If posix_spawn can't be used, implement a sidecar "spawner" process that pageserver asks to spawn walredo processes.

• Option 1: extend the existing walredo C code to enter "template" mode.
• Option 2: fork off a pagserver child process that will act as the spawner process

NB: we decide against a pool of pre-spawned walredo processes as the amoutn of CPU wasted on the inefficient fork() call is significant.

Background Reading

• https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/
• https://lwn.net/Articles/908915/
• https://lwn.net/Articles/908268/

Work

Solve The Issue

Beta Give feedback

1. feat(walredo): various observability improvements #6573
   +0
2. feat(walredo): use posix_spawn by moving close_fds() work to walredo C code #6574
   +0
3. walredo: regression test that will ensure we use posix_spawn #6630
   +0
4. measure impact in staging & prod => merge above preliminary work to get better observability
5. it's good, we wrote a blog post about it

Follow-Ups

Beta Give feedback

1. Portability: Close file descriptors in walredo process #6580
   +0
   

Spin-Offs (no need to complete before closing)

• Epic: fully eliminate latency impact of walredo spawning #6581
• walredo: eliminate the need for close_range #6646

[jcsp]

• walredo fork changes will address basebackup pain
• but spawn still takes tens of millis, so first getpage request in a while from a running database, that is a latency spike -- hence motivation to use a pool.