fix(rustyclawd): replace Bash-tool wall-clock cap with idle-liveness reaping (#2607)#2650
Merged
rysweet merged 1 commit intoJul 6, 2026
Conversation
…reaping (#2607) The RustyClawd adapter's `Bash` tool wrapped `child.wait_with_output()` in a `tokio::time::timeout(120s, …)` and spawned without isolation. That wall-clock cap SIGKILLed a still-producing command at 120s and, because the child was neither process-group isolated nor signalled on drop, orphaned it — exactly the class of bug issue #2607 targets. Replace the cap with idle-liveness, mirroring `meeting_backend::agent_proxy`: - Spawn via `ProcessSpawnConfig::with_isolation()` (setsid) so the child leads its own process group (pid == pgid) and the reaper can kill the whole subtree. - Stream stdout/stderr over a channel; every chunk resets `last_activity`. A command that keeps producing output runs unbounded regardless of total runtime. Only sustained silence for the whole window reaps the child, and the error honestly identifies an IDLE reap (not a productive kill). - New `SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS` (seconds) sets the window; `0` disables reaping entirely (fully unbounded escape hatch). Unset/malformed falls back to the model-supplied per-call `timeout` (ms, default 120s), now reinterpreted as an idle window rather than a total budget. Tests (the three #2607 acceptance scenarios) plus pure resolver cases: - a command emitting output past the window is never killed; - a genuinely idle child is reaped after the window with an honest error and no orphaned process group survives; - `SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS=0` disables reaping. Docs: new reference `rustyclawd-bash-tool-idle-liveness.md` (behavior, config, mechanism, and the full audit of every wall-clock cap on agent/LLM work, classified keep/replace), linked from the index and base-type-adapters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4 tasks
📊 Coverage Summary
Coverage data from CI run. Test files matching |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #2607 — no wall-clock timeout may kill in-flight agent/LLM work.
The #2607 audit swept every
tokio::time::timeout/wait_timeout/recv_timeouton a child/agent/LLM path. Exactly one remaining wall-clock kill on local agentic work was found: the RustyClawd adapter'sBashtool. This PR replaces it with idle-liveness, mirroring the canonical meeting agent proxy (src/meeting_backend/agent_proxy.rs).The bug
The
Basharm wrappedchild.wait_with_output()intokio::time::timeout(120s, …)and spawned withProcessSpawnConfig::default()(no isolation). A command still streaming output at 120 s was SIGKILLed mid-flight →ClientError::Timeout. Worse, because the child was neither process-group isolated nor signalled on future-drop, it was orphaned — left running detached while the tool call reported failure.The fix (idle-liveness)
ProcessSpawnConfig::with_isolation()(setsid): the child leads its own process group (pid == pgid) so the reaper can SIGKILL the whole subtree.last_activity. A command that keeps producing output runs unbounded regardless of total runtime. Only sustained silence for the whole window reaps the child, killing the whole process group (numeric-PIDlibc::kill(-pgid, SIGKILL), shell-free signal policy) so no orphan survives. The error honestly identifies an IDLE reap, never a productive kill.SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS(seconds) sets the window;0= fully unbounded escape hatch. Unset/malformed falls back to the model-supplied per-calltimeout(ms, default 120 s), now reinterpreted as an idle window rather than a total budget.Tests (the three #2607 acceptance scenarios + resolver)
bash_producing_output_past_the_window_is_never_killed_2607— a command emitting output past the window runs to completion, all output captured.bash_idle_child_is_reaped_with_honest_error_and_no_orphan_2607— a genuinely idle child is reaped after the window with an honest "idle" error, and no PID retains the command's marker (whole process group reaped, no orphan).bash_env_zero_disables_idle_reaping_2607—SIMARD_RUSTYCLAWD_IDLE_LIVENESS_SECS=0disables reaping.resolve_idle_windowcases: positive override wins,0→None, unset/malformed → per-call fallback.Audit — wall-clock caps on agent work (keep vs replace)
base_type_rustyclawd/tool_executor.rs— Bash armmeeting_backend/agent_proxy.rs— idle window0= unbounded)meeting_backend/agent_proxy.rs—recv_timeout(200 ms)signal_conversation/channel.rs—signal_agent_mode()engineer_loop/agent_spawn.rs—wait_with_output()agent_supervisorheartbeat stalenessterminal_sessionidle detectionmeeting_backend/close_guard.rscopilot_task_submit/orchestration.rsoperator_commands_dashboard/*bridge_subprocess/subprocess.rs,update_check.rsKeeps are deliberate. The rule targets agent turns and LLM subprocesses; a remote-job poll, a channel-framing read, or a graceful-shutdown budget is legitimately bounded and stays bounded.
Docs
New reference
docs/reference/rustyclawd-bash-tool-idle-liveness.md(behavior, configuration, mechanism, full audit, invariants), linked fromdocs/index.mdanddocs/reference/base-type-adapters.md.mkdocs build --strictpasses.Verification
cargo test --lib base_type_rustyclawd::tool_executor— 24 passed.cargo clippy --release -- -D warningsand full--all-targets --all-features --lockedclippy — clean.cargo fmt --all -- --check— clean.mkdocs build --strict— clean.origin/main; diff is exactly the 5 intended files.Closes #2607
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com