FE-1220: Harden and document Petrinaut optimization execution isolation#9058
FE-1220: Harden and document Petrinaut optimization execution isolation#9058kube wants to merge 3 commits into
Conversation
…lation latency NodeAPI (apps/hash-api): - Race the backpressure wait against drain, close, and the request lifecycle abort signal, removing the losing listeners via an internal AbortController so stalled writes cannot leak listeners, strand a rejected promise, or hold an optimization slot past the timeouts. - Mark terminal events as committed at write hand-off so an abort during the final backpressured write cannot append a second, contradictory terminal event. - Write the failure-path terminal event without awaiting backpressure so teardown never blocks on a stalled client. - Skip transport heartbeats while the response buffer needs draining. Petrinaut Optimizer (apps/petrinaut-opt): - PetrinautModel.close(graceful=False) signals the CLI process group immediately instead of waiting up to five seconds for stdin EOF; the graceful EOF wait is now reserved for studies that completed normally. All failure, timeout, and cancellation paths terminate promptly. - Share one idempotent per-run cleanup between the stream wrapper and a StreamingResponse background task, so a client that aborts before the body is ever pulled cannot leak the admission slot or a live CLI. - Close an abandoned CLI via an initializer done-callback when a second cancellation races the init-cancel recovery. - Emit Retry-After on the 429 study-limit response and declare it in the OpenAPI spec. Tests cover drain-first, close-first, client abort, idle timeout during backpressure, single-terminal-event, heartbeat suppression, slot release, prompt process-group termination, and 429/Retry-After propagation across NodeAPI, the optimizer client, and the Python service.
Petrinaut Optimizer (apps/petrinaut-opt): - Apply resource.prlimit bounds (Linux) to every spawned CLI process: RLIMIT_CPU, RLIMIT_AS, and RLIMIT_NPROC, each configurable via HASH_PETRINAUT_OPT_CLI_* environment variables with zero disabling. - Sweep the CLI's process group with SIGKILL at the end of every close path, so descendants that remain in the group cannot outlive the study even when the CLI itself exits gracefully on stdin EOF. - Enforce a per-study wall-clock ceiling (HASH_PETRINAUT_OPT_MAX_STUDY_SECONDS, default 900 s) in both stream loops, emitting a terminal error frame and closing the CLI promptly; a study whose terminal sentinel is already queued when the deadline lapses is still reported as completed. - Run tini as PID 1 in the Docker image so orphaned descendants are reaped instead of accumulating as zombies. Documentation: - New apps/petrinaut-opt/docs/threat-model.md: execution-path threat model, per-layer mitigations, known limitations (per-UID RLIMIT_NPROC accounting, setsid escape from the group sweep, prlimit timing, pgid recycling), required out-of-repo deployment configuration (read-only rootfs, task memory sized against RLIMIT_AS, deny-all egress, pidsLimit), and the decision that no additional V8 sandbox is needed. - README documents the new environment variables; the CLI's OPTIMIZATION_INTEGRATION.md documents the cancellation and shutdown contract. Tests cover the grandchild sweep (real subprocess, mutation-verified), prompt-vs-graceful shutdown ordering, rlimit application and parsing, ceiling-based termination, and the completed-at-the-boundary case. Real subprocess tests pin the limits off so per-UID NPROC accounting on busy hosts cannot fail them.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
github-actions
Bot
added
area/infra
PR SummaryMedium Risk Overview Runtime hardening ( Documentation: new Tests cover group sweep, rlimit application/parsing, study ceiling vs. boundary completion, and disabling limits in real-subprocess tests. Reviewed by Cursor Bugbot for commit 47f2fea. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 47f2fea. Configure here.
| f"{json.dumps({'state': 'ERROR', 'message': message})}" | ||
| f"\n\n" | ||
| ) | ||
| break |
There was a problem hiding this comment.
Pending sentinel misclassified as timeout
Medium Severity
When the study wall-clock deadline is reached, timeout handling treats an empty events queue as proof the study is still running. The worker always enqueues the completion sentinel via call_soon_threadsafe, so a study that just finished can look idle until that callback runs, triggering a false execution-limit error instead of the normal done path.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 47f2fea. Configure here.
|
Converted back to draft: the stack was reordered so the streaming fixes (#9057) and correlated logging (#9059) can merge first to debug the staging |
github-actions
Bot
added
area/apps > hash*
2 participants
🌟 What is the purpose of this PR?
Hardens and documents the isolation of the Petrinaut optimization execution path (NodeAPI → optimizer → CLI). Threat-models the path, terminates the complete CLI process group on every shutdown, bounds CLI CPU / memory / process count, adds a study wall-clock ceiling, and reaps orphans — with the deployment isolation model and remaining risks written down.
🔗 Related links
🚫 Blocked by
🔍 What does this change?
petrinaut_client.py: aSIGKILLsweep of the CLI process group on every close path, so descendants cannot outlive the study;resource.prlimit(Linux) bounds each CLI process —RLIMIT_CPU/RLIMIT_AS/RLIMIT_NPROC, configurable viaHASH_PETRINAUT_OPT_CLI_*(0 disables).petrinaut_optimizer.py: a study wall-clock ceiling (HASH_PETRINAUT_OPT_MAX_STUDY_SECONDS, default 900) that terminates the stream and the CLI.tinias PID 1 to reap any orphan that escapes the group sweep.docs/threat-model.md(new): the execution path, per-layer mitigations, required out-of-repo deployment configuration, and the documented decision that container/process isolation is sufficient (no additional V8 sandbox required). README andOPTIMIZATION_INTEGRATION.mddocument the new env vars and the cancellation/shutdown contract.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
RLIMIT_NPROCis enforced by Linux against the service user's total task count, not this child's own descendants; container-levelpidsLimit(SRE-830) is the backstop.prlimitis applied from the parent immediately after spawn, so the child is briefly unbounded; container limits are the outer backstop.🛡 What tests cover this?
New
apps/petrinaut-optunit tests: real-subprocess grandchild process-group sweep (mutation-verified), resource-limit application and env parsing, and study wall-clock-ceiling termination.❓ How to test this?
cd apps/petrinaut-opt && uv run pytestprlimit; it is skipped on macOS).