Replace test-count recycling with resource-based worker limits

[thejchap]

Replace the fixed MAX_TESTS_PER_WORKER constant with a flexible, resource-aware worker recycling system that monitors memory usage, open file descriptors, and wall-clock age.

Summary

Worker processes are now recycled based on soft resource ceilings rather than a hard test count. This allows the runner to respond to actual resource pressure (memory leaks, FD accumulation) while remaining deterministic and testable.

Key Changes

• New resource monitoring types (WorkerHealth, WorkerLimits, RecycleReason):

  • WorkerHealth: Captures worker self-reported RSS bytes and open FD count
  • WorkerLimits: Configurable soft ceilings (default: 1 GiB RSS, 200 FDs, 10 min age)
  • RecycleReason: Enum indicating which limit triggered recycling, with human-readable Display impl for debug logs

• Pure recycle-decision logic (evaluate_recycle function):

  • Factored out of WorkerProcess for unit testability
  • Checks signals in priority order (memory > FDs > age) to report the strongest reason
  • Respects None limits (opt-out) and missing readings (platform unavailability)

• Worker-side health reporting (Python):

  • New _measure_rss_bytes() and _measure_open_fds() functions using resource module and /proc//dev filesystem
  • _measure_health() snapshot attached to every run_test/run_doctest response
  • Platform-aware: returns None for unavailable signals rather than guessing

• Protocol updates:

  • New WorkerHealthWire struct for wire format
  • New RunTestResponseWire wrapper combining health snapshot with test result
  • Python side merges outcome dict with top-level health field

• Pool API expansion:

  • New WorkerPool::with_python_path_and_limits() for tests to set custom limits
  • WorkerSpawnCtx struct bundles spawn-time parameters to reduce signature bloat
  • Recycling now deferred to end-of-unit (after finalize hooks) to preserve scope fixture teardown

• Test updates:

  • worker_recycles_after_max_tests → worker_recycles_when_age_exceeds_limit: Uses tiny max_age cap and sleep to drive deterministic recycling
  • recycle_does_not_skip_scope_fixture_teardown: Simplified to use age-based recycling within a single unit

Notable Implementation Details

• Recycling is checked at end-of-unit (not mid-test) to ensure per="scope" fixture teardown runs before process death
• Health snapshot is updated after every test response, enabling responsive recycling
• Default limits are tuned for real workloads: 1 GiB RSS (headroom for heavy native deps), 200 FDs (below macOS soft limit), 10 min age (bounds slow leaks)
• WorkerLimits::unlimited() helper disables all caps for tests exercising one signal in isolation
• Recycling reason is logged for post-mortem triage of memory leaks vs FD pressure vs age drift

https://claude.ai/code/session_01PMbxzSuASTEYbDFEu4SQqy