Fix up the process orchestrator

[benesch]

There are two known problems with the process orchestrator:

• It can lose track of processes, possibly leading to sources that get "stuck" in CI but not in production (see storage: sources can get silently stuck after restart #15155 (comment)).
• It can't keep ownership of TCP ports in case of process restarts (code). test: Fix process orchestrator port re-use #15336

We've talked about simplifying the process orchestrator (e.g., removing all support for re-adopting processes after an environmentd restart), but there are at least two good reasons to want to keep the process orchestrator in parity with the Kubernetes orchestrator:

• The process orchestrator is incredibly useful for local development and manual testing. It was crucial to the efficient development of storage and compute reconciliation, where you want to be able to restart envd and have it adopt existing storaged and computed processes.
• If we more widely advertise the all-in-one materialized Docker image, we'll want a robust process orchestrator that we trust.

I think there are three well-scoped improvements we can make.

1. Use Unix domain sockets rather than ports. This will eliminate just an absolute ton of complexity from the process orchestrator. TCP ports are a scarce, global resource; Unix domain sockets are not scarce and easily scoped to a per-test/per-envd-invocation directory. This will be a fairly large refactor of our networking/orchestration layer, but a net reduction in complexity. Added benefits: it'll be possible to run bin/sqllogictest and bin/environmentd at the same time. It'll be possible to use cargo-nextest to run Rust tests faster (ci: pull cargo test out of the build CI job #13035).
2. Figure out how to supervise adopted processes. Right now the process orchestrator will restart processes that it spawns if they crash... but AFAICT it will not restart processes that it adopts. There's just no call to the equivalent of supervise that watches for the external process to crash.
3. Write down a "service spec" in the PID file, rather than just a PID, and kill the existing process if the service spec on disk does not match the desired service spec. Right now we assume that the existing process was spawned with all the right arguments, which is not a valid assumption in general—e.g., if you wanted to test a version upgrade with the process orchestrator.

Tossing this one on @guswynn and @jkosh44's radar, but the process orchestrator isn't really owned by any particular team.

cc @chaas @uce

[philip-stoev]

This bug actually prevents running any tests with SIZE '4-4' reliably in the CI, let alone the entire CI. The sheer number of processes that need to be started properly is such that sporadic failures are guaranteed to spoil the CI experience for everyone. Both the "storaged was never restarted" and the "cannot assign requested address" sub-issues are seen.

@benesch could this be a potential solution:

• put all the metadata/PID files on ephimeral storage
• if a container restarts, this information will be lost , but all the children processes are dead too, so we have effectively a clean-slate state from which we should be able to respawn all the required children?

[benesch]

put all the metadata/PID files on ephimeral storage

Yup, I think that would do it!

[guswynn]

@benesch is ephemeral storage the same as anonymous volumes in docker compose or is even even more ephemeral?

[benesch]

Even more ephemeral! Anonymous volumes are still volumes that can persist across container restarts. Ephemeral storage is the stuff that vanishes on container exit. Like, writing to /home, unless you've mounted a volume at /home, for example.