Rethinking cgroup Memory Enforcement

[josephjclark]

Re-implementation of #1467

This rework takes a different approach to cgroup enforcement than #1467.

#1467 had the engine self-provision the whole cgroup hierarchy, starting from
the real cgroup root (/sys/fs/cgroup) and building a fixed path down from
there. That requires root, and if the root cgroup is already populated (which
it always is), it relocates every process on the host/container into a leader
cgroup just to make room — a much bigger footprint than we need.

This branch instead has the engine just use whatever cgroup it's already
running in (read from /proc/self/cgroup). No root needed, no touching the
host root — we rely on the environment (container runtime, systemd Delegate=,
etc.) to have already put us somewhere writable, and only create our per-run
leaves under that.

Other simplifications that came out of this:

• Cgroup enforcement is now off by default, and fully separate from the V8
  heap limit (--max-old-space-size) — you can toggle either independently.
• No more separate "cgroup memory limit" and "cgroup parent" options — the
  ceiling is just derived from the existing run-memory setting, and there's
  no legitimate case where you'd want the leaf cgroup created anywhere other
  than wherever the engine already lives.
• ws-worker gets a single --cgroup flag instead of two separate
  options.
• The enforcement test is gated on actually detecting a writable cgroup
  (not just "is this Linux"), and is excluded from the normal test run since
  it's only safe inside an isolated cgroup — running it in an ordinary dev
  shell can take down more than just the test.

Claude notes:

What changed
cgroup.ts — the core rework. ensureParent and all root-hierarchy mutation deleted. New resolveSelfCgroup() reads /proc/self/cgroup to default the parent to the cgroup the worker was started in. detect() is now read-only (exists, under the mount, memory controller present, writable); prepare() delegates the memory controller with the leader move confined strictly to the given parent. Also: the rmdir-retry timer is unref'd.

pool.ts — default parent is now resolveSelfCgroup().

lifecycle.ts / events.ts — source is now forwarded on WORKFLOW_ERROR, closing the propagation gap from my review. One discovery here: EngineError already had an unused source = 'engine' field, which the PR's 'heap' | 'cgroup' silently overloads.

[josephjclark]

Rename procs to Processes

[josephjclark]

The cgroup limit should be derived from the memoryLimit, no? Maybe we accept a multiplier or overhead as an argument, but I think this should be automated

[josephjclark]

@doc-han I'd appreciate you spending an hour with this and letting me know what you think. There are pretty good docs (I hope!) in the worker and engine readmes

The outstanding gotcha is permissions. This will only work if the docker image runs with --privileged right now (I presume this is the same change we need in your original PR?). I've opened a thread in deployment to try and work out a solution: https://github.com/OpenFn/deployment/issues/482