Skip to content

Agent guidance gap: completion detection for backgrounded codex/CLI delegation (pgrep self-match trap) #863

Description

@arul28

What happened

During the web-client/relay perf lane (2026-07-20), the Work-chat agent launched five codex exec reviewers detached (codex exec … & inside one Bash call) and then spawned background waiter loops using until ! pgrep -f "codex exec -s read-only"; do sleep 30; done.

pgrep -f matches full command lines — and the waiter's own zsh -c wrapper contains that exact string, so each waiter matched itself and spun forever. The reviews finished at 03:50; the agent discovered them at 11:09 only because the user pinged. ~7 hours of dead air on an otherwise-healthy pipeline.

Root causes

  1. Detach-then-poll instead of harness-native notification. When a command is launched as its own run_in_background Bash task, the harness fires a completion notification on process exit — no waiter needed. Detaching with & inside a foreground call forfeits that and forces hand-rolled polling.
  2. pgrep -f self-match trap. Any waiter whose own command line contains the search pattern never terminates. Classic, but nothing in the bundled guidance warns about it.
  3. Existing guidance covers launching, not finishing. The project CLAUDE.md delegation section is good on how to launch codex (</dev/null, file log, session-file check for wedge detection) but silent on how to detect completion. A prior memory note ("bg Bash pollers unreliable in Work chats — use one-shot foreground checks") existed but wasn't reinforced anywhere discoverable.
  4. ADE's sanctioned durable-wait primitives (ScheduleWakeup mirrored into the durable scheduler, ade chat scheduled-work create) were not used — partly because no skill connects "waiting on a delegated CLI run" to those primitives.

Recommendations

Docs/skill (cheap, high value): add a "Waiting for delegated runs" section to the delegation guidance (CLAUDE.md model-delegation section and/or an ade-bundled agent skill), stating:

  • Prefer one run_in_background Bash call per delegated CLI run — the harness notifies on exit. Launch N runs as N background tasks, not one &-fan-out.
  • If a waiter loop is unavoidable: wait on the PID (kill -0 $PID) or a sentinel file (codex exec …; touch /tmp/run.done). Never pgrep -f with a pattern that could appear in the waiter's own command line (quote the trap explicitly).
  • Log-marker waits must account for the prompt being echoed into the log (marker can match the echoed prompt — require the marker after the echo, or count occurrences, or use a sentinel file instead).
  • For long or cross-restart waits, use ScheduleWakeup (ADE-durable) or ade chat scheduled-work create as the fallback heartbeat, with a stated reason.

Possible CLI affordance (optional): an ade run --background --notify style wrapper (or documented pattern) that executes an arbitrary command, writes a completion sentinel + exit code to a known path, and surfaces it in ade actions/chat notifications — so agents never hand-roll process-watching for provider CLIs that ADE doesn't natively track.

Acceptance

  • Bundled skill / CLAUDE.md delegation section documents the completion-detection patterns and the two traps (pgrep self-match, prompt-echo marker match).
  • At least one worked example: N parallel codex exec runs with harness-native completion notifications.

Filed by the Work-chat agent at the user's request after the incident; context lane: ade/codex-auth-login-button-fd3c9bbe.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions