agent-watch — telling DONE, FAILED and STALL apart when you run codex exec in the background

[soul-sol]

When I started running codex exec in the background for parallel work, the thing that cost me the most time wasn't a bad diff — it was not knowing whether a worker was done, dead, or just stuck.

A background worker that silently stopped to ask for approval looks exactly like one that is still thinking. And a worker that crashed looks exactly like one that finished, if the only thing you check is that the process is gone.

So I wrote two small POSIX scripts that answer that question honestly, and open-sourced them (MIT): https://github.com/soul-sol/agent-watch

Four states, not two

RUNNING api_task pid=41233
DONE    api_task — codex marker found
FAILED  api_task — exit=1
STALL   api_task — process exited without a completion conclusion

STALL is the state I kept missing. The process is gone, the recorded exit code is 0, and yet the worker never actually concluded — it stopped to ask something, or died mid-thought. Treating that as "done" is how you lose an afternoon.

Usage

# launch — records .log, .pid and .exit
./worker_launch.sh api_task ./logs -- codex exec --skip-git-repo-check -C "$PWD" \
  -m <MODEL> -s workspace-write "TASK: fix the billing status mapping. VERIFY: npm test -- billing. REPORT: end with DONE: <summary>."

# ask what happened, any time
./worker_watch.sh api_task "$(cat logs/api_task.pid)" logs/api_task.log codex logs/api_task.exit

And a whole wave at once:

for w in api ui migration; do
  ./worker_watch.sh "$w" "$(cat logs/$w.pid)" "logs/$w.log" other "logs/$w.exit"
done

Three details that turned out to matter

1. Record the exit code to a file. A background process's status is gone the moment you stop waiting on it, so worker_launch.sh writes $? into <name>.exit. Without that, you cannot distinguish "finished cleanly" from "crashed" after the fact.

2. Look for completion markers only in the log tail. codex prints a token-usage line when it finishes normally, which makes a great marker — until the worker reads a document that mentions that phrase and echoes it mid-run. Matching anywhere in the file gives false DONEs; matching in the last 40 lines does not. (I got burned by exactly this.)

3. Markers differ per tool. codex has one; claude -p and gemini-family CLIs don't, so for those the honest test is "process exited AND the log tail contains a conclusion". The script takes codex or other as an argument rather than pretending one rule fits all.

Prompt-side half

The scripts only work if the worker is told to produce a terminal state. Two lines in the brief do most of the work:

EXECUTION RULE: Do not ask questions; proceed immediately with reasonable defaults and report your assumptions when finished. Do not wait for approval.
REPORT: end your output with `DONE: <one-line summary>` or `FAILED: <reason>`.

Without the first line, codex will happily stop and wait for a human who isn't watching. That single sentence eliminated most of my stalls.

No dependencies beyond a POSIX shell and coreutils. Feedback and PRs welcome — especially on markers for CLIs I don't use.

[ooocooc]

Nice separation of process state, recorded exit code, and a completion conclusion. One Codex-specific hardening looks worthwhile: current codex exec --json emits structured JSONL events, including terminal turn.completed, turn.failed, and error events. Parsing those event types would be less fragile than matching the human-facing tokens used line. Tail-only matching narrows false positives, but a retrieved document or the final answer can still contain that exact phrase within the last 40 lines. I would add fixtures for that case and for a truncated JSONL stream, while keeping the text marker as a fallback for older Codex versions. Would a separate codex-json kind fit your goal of keeping the default scripts small and dependency-free?

[soul-sol]

You are right, and I shipped it: v0.2.0 adds a codex-json kind that parses the JSONL terminal events instead of prose.

I checked the current event stream and the terminal shapes are exactly as you described:

{"type":"thread.started"}
{"type":"turn.started"}
{"type":"item.completed","item":{...}}
{"type":"turn.completed","usage":{...}}

So the watcher now takes the last turn.completed / turn.failed in the tail as authoritative, treats a "type":"error" event as FAILED, and only falls through to STALL when no terminal event is present:

last_event=$(tail -n 200 "$log" | grep -oE '"type":"turn\.(completed|failed)"' | tail -n 1)

To your question — yes, a separate kind was the right call rather than changing codex. It keeps the scripts dependency-free (no jq: the event type is matched as a literal string, and taking the last match handles a truncated or interleaved stream), and the old text-marker path still works for people on older Codex versions who are not passing --json. Usage is now:

./worker_launch.sh api_task ./logs -- codex exec --json --skip-git-repo-check -C "$PWD" -m <MODEL> -s workspace-write "<brief>"
./worker_watch.sh api_task "$(cat logs/api_task.pid)" logs/api_task.log codex-json logs/api_task.exit

Your point about the phrase appearing within the last 40 lines is the real hole — tail-only narrows it but never closes it, which is exactly why the structured path had to exist. Fixtures for the echoed-marker case and a truncated JSONL stream are the next thing I will add.

Thanks for the sharp review: https://github.com/soul-sol/agent-watch/releases/tag/v0.2.0