Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/harness-adapters/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ The false-wedge exposure is real and was reproduced live (docs/codex-busy-detect
The busy-signature matching in `fm-watch.sh` and `fm-tmux-lib.sh` stays deliberately unchanged - the row still matches whenever it renders - but the absorb path now carries an interface-text-INDEPENDENT backstop scoped to codex.
`codex_static_pane_upgrade` in `bin/fm-watch.sh` treats an otherwise-stale codex pane as provably working (absorb plus wedge timer) whenever the codex agent PROCESS is confidently alive: `fm_backend_agent_alive` reads `alive` because the pane's foreground command stays `codex` for the entire turn (verified 100/100 samples), a signal that does not depend on the busy row at all.
A crashed codex reads `dead` (bare shell) and still surfaces at once, and a genuinely wedged codex still escalates past `STALE_ESCALATE_SECS` and on to demand-deep-inspection, so the backstop hides a healthy worker without hiding a stuck one.
One stale pane never reaches the backstop at all: when `bin/fm-crew-state.sh`'s run-step authoritatively reports the run stopped at a no-mistakes decision gate, that pane surfaces on first sight for every harness alike, so codex gets no first-sight absorb the others lack (docs/architecture.md owns the gate rule).
This also covers the keymap-remap blind spot below, because it does not read the interrupt-hint text.

The keymap case is the one that requires an operator to have changed something.
Expand Down
25 changes: 21 additions & 4 deletions bin/fm-classify-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -363,15 +363,29 @@ signal_reason_is_actionable() { # <file> ...
# (e.g. waiting on CI);
# paused - the crew's authoritative current state is a declared external-wait
# pause (paused:), which is EXPECTED to idle;
# parked - the crew's authoritative RUN-STEP state is a decision gate
# (no-mistakes awaiting_approval/fix_review). The pipeline stopped on
# purpose and is waiting for an answer, so an idle pane is the
# CORRECT behavior rather than a wedge. Deliberately restricted to
# source: run-step - a status-log-sourced `parked` is the same
# append-only needs-decision EVENT the stale path already reasons
# about, is arbitrarily old, and reads identically whether the crew is
# at the gate or died an hour ago.
# This token says the RUN is parked; it says NOTHING about the worker
# still being there, because a crashed worker leaves the run parked
# forever. A caller must corroborate with an interface-text-
# independent liveness signal before absorbing on it
# (bin/fm-watch.sh's parked_gate_liveness_class);
# degraded- the reader could not consult its authoritative source at all: a
# required tool is absent, or the call it makes could not be made or
# did not answer (fm-crew-state.sh's `degraded` verdict, whose cause
# token says which). NOT a statement about the crew: the supervisor
# must report the broken instrument rather than draw either
# conclusion from it;
# none - neither, so the wake must surface (a stopped/finished/parked/failed/
# torn-down/unknown crew, or an unreadable verdict).
# One fm-crew-state.sh read serves BOTH absorb reasons at once. Reading the state
# none - none of the above, so the wake must surface (a stopped/finished/
# failed/torn-down/unknown crew, a gate reported only by the status
# log, or an unreadable verdict).
# One fm-crew-state.sh read serves EVERY absorb reason at once. Reading the state
# authoritatively (not the status log) is what keeps run-step precedence: a crew
# that appended paused: but then STARTED a run reports working, never paused.
# NOT a pure read: fm-crew-state.sh may make a bounded no-mistakes call, so callers
Expand All @@ -386,10 +400,13 @@ crew_absorb_class() { # <id>
state=${line#state: }; state=${state%% *}
if [ "$state" = paused ]; then printf 'paused'; return; fi
if [ "$state" = degraded ]; then printf 'degraded'; return; fi
src=${line#*source: }; src=${src%% *}
if [ "$state" = working ]; then
src=${line#*source: }; src=${src%% *}
case "$src" in run-step|pane) printf 'working'; return ;; esac
fi
if [ "$state" = parked ]; then
case "$src" in run-step) printf 'parked'; return ;; esac
fi
printf 'none'
}

Expand Down
5 changes: 5 additions & 0 deletions bin/fm-mark-parked.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
# terminal outcome has been relayed and only external human action remains.
# bin/fm-watch.sh owns window validation, secondmate rejection, key derivation,
# and marker creation; this wrapper only provides a seatbelt-safe command shape.
# NOT for a mid-pipeline worker waiting at a no-mistakes decision gate: that one
# is neither terminal nor waiting on anything external, the watcher recognizes it
# on its own from the authoritative run-step plus agent liveness
# (bin/fm-watch.sh's parked_gate_liveness_class), and a hand-placed marker here
# would keep muting that pane after the gate was answered.
set -u

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
Expand Down
167 changes: 143 additions & 24 deletions bin/fm-watch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,14 @@
# otherwise, so a crew that finishes (or stops and waits) without a current
# working signal is never silently swallowed. A declared external-wait pause or
# a firstmate-declared parked terminal task is the separate idle absorb case and
# re-surfaces only on its long bounded cadence. A new status write still surfaces
# immediately in normal mode and clears parked tracking.
# re-surfaces only on its long bounded cadence. A run the authoritative run-step
# reports stopped at a DECISION GATE still surfaces on first sight - nothing has
# relayed that gate yet - but does not ESCALATE as a wedge afterwards, since
# idling at a gate is the correct behavior; that hold lasts only as long as the
# worker's agent PROCESS is confirmed alive, because a crashed worker leaves the
# run parked identically and forever (parked_gate_liveness_class).
# A new status write still surfaces immediately in normal mode and clears parked
# tracking.
# While state/.afk exists, the away daemon owns triage and this watcher queues
# every actionable wake without running the more expensive normal-mode
# classifiers, though signal records are collapsed to one per task in away mode
Expand All @@ -26,8 +32,11 @@
# firstmate hands it to a no-mistakes validation. A declared
# external-wait pause or firstmate-declared parked terminal
# wait is absorbed instead with its own long re-surface
# cadence, never as a wedge. Only when neither absorb class
# applies does the log's last line decide:
# cadence, never as a wedge. A run parked at a decision gate
# whose worker is confirmed alive surfaces its first sighting
# like any other stopped crew, then holds the wedge ladder on
# the same bounded-recheck terms as an active run. Only when
# no absorb class applies does the log's last line decide:
# terminal (captain-relevant) or non-terminal (no verb),
# both surfaced at once. A provably-working stale past the
# wedge threshold also surfaces, with an "escalation N"
Expand Down Expand Up @@ -358,6 +367,32 @@ wake() {
# below).
FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3}

# ladder_hold: the one implementation of "this stale pane has a positive reason not
# to climb the wedge ladder". Refreshes the escalation timer so the reason is
# RE-READ every window rather than trusted once, and surfaces one bounded recheck
# whenever the pane has been frozen for a full PAUSE_RESURFACE_SECS - measured on
# the frozen hash's own age, not on the timer this hold keeps refreshing, so a hold
# can never postpone its own recheck. It never touches the escalation counter, so a
# held pane never climbs toward demand-deep-inspection.
# <situation> names the evidence in the wake text and the triage log; <next-step>
# is what firstmate should confirm. Both callers pass their own, because "the run
# is running" and "the run is parked at a gate" need different confirmations.
ladder_hold() { # <window> <since-file> <triage-label> <situation> <next-step>
local win=$1 since_file=$2 label=$3 situation=$4 next_step=$5 wkey hold_age rf reason
date +%s > "$since_file"
wkey=$(window_state_key "$win")
hold_age=$(age_of "$STATE/.stale-$wkey")
rf="$STATE/.wedgeheld-$wkey"
if [ "$hold_age" -ge "$PAUSE_RESURFACE_SECS" ] && [ "$(age_of "$rf")" -ge "$PAUSE_RESURFACE_SECS" ]; then
reason="stale: $win (pane unchanged ${hold_age}s while $situation - bounded recheck on a long cadence, not a wedge escalation; $next_step)"
fm_wake_append stale "$win" "$reason" || exit 1
date +%s > "$rf"
wake "$reason"
return
fi
triage_log "absorbed $label ($situation, ladder held): $win"
}

# Repeat-poll wedge-timer bookkeeping for an already-classified stale hash
# absorbed as provably-working - repairs a missing/corrupt timer (self-heals a
# watcher restart between recording the hash and recording the timer), or
Expand All @@ -378,8 +413,13 @@ FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3}
# read is what distinguishes them, it runs once per interval rather than once per
# poll, and the moment a run ends or dies the crew stops reading as working and
# the next elapse escalates on the unchanged schedule.
#
# A run PARKED at a decision gate holds the ladder on the same terms, with one
# extra requirement the active-run case does not need: the gate reading is only
# trusted while agent liveness confirms a worker is still there to answer it
# (parked_gate_liveness_class). Both holds are one function - ladder_hold above.
wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-file>
local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason wkey hold_age rf wtask wclass
local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason wtask wclass
since=$(cat "$since_file" 2>/dev/null || true)
case "$since" in
''|*[!0-9]*)
Expand All @@ -391,6 +431,15 @@ wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-
if [ "$age" -ge "$STALE_ESCALATE_SECS" ]; then
wtask=$(window_to_task "$win" "$STATE")
wclass=$(crew_absorb_class "$wtask")
# The gate reading is re-corroborated against agent liveness HERE, not
# carried over from the classification that started this timer: a worker
# can die at any point during the hold, and a stale `alive` from minutes
# ago is exactly the evidence that would let it rot. A dead or unreadable
# agent falls straight through to the ordinary escalation below, on the
# unchanged timings. The codex static-pane upgrade is deliberately NOT
# applied here - it absorbs a first sighting only, and a genuinely wedged
# codex must still reach demand-deep-inspection.
[ "$wclass" = parked ] && wclass=$(parked_gate_liveness_class "$win")
if [ "$wclass" = degraded ]; then
# The crew state was never read - a tool the reader needs is missing
# from this service's PATH. Climbing the ladder here would escalate a
Expand All @@ -402,27 +451,24 @@ wedge_timer_check() { # <window> <since-file> <triage-label> <escalation-count-
return
fi
if [ "$wclass" = working ]; then
date +%s > "$since_file"
# Bounded insurance against the one way an active run can lie: a run
# whose agent died mid-step keeps reporting `running` indefinitely, and
# holding the ladder on that reading alone would let a dead crew rot
# invisibly. So a hold that has lasted a full PAUSE_RESURFACE_SECS - the
# fleet's existing "recheck a known-idle wait" cadence, measured on the
# frozen pane hash's own age, not on the timer this hold keeps
# refreshing - surfaces ONE bounded recheck and then waits out another
# full window. It never touches the escalation counter, so an active run
# still never climbs the wedge ladder toward demand-deep-inspection.
wkey=$(window_state_key "$win")
hold_age=$(age_of "$STATE/.stale-$wkey")
rf="$STATE/.wedgeheld-$wkey"
if [ "$hold_age" -ge "$PAUSE_RESURFACE_SECS" ] && [ "$(age_of "$rf")" -ge "$PAUSE_RESURFACE_SECS" ]; then
reason="stale: $win (pane unchanged ${hold_age}s while the run still reports active - bounded recheck on a long cadence, not a wedge escalation; confirm the run is really progressing)"
fm_wake_append stale "$win" "$reason" || exit 1
date +%s > "$rf"
wake "$reason"
return
fi
triage_log "absorbed $label (run still active at escalation, ladder held): $win"
# invisibly - so the hold earns one bounded recheck per window.
ladder_hold "$win" "$since_file" "$label" \
"the run still reports active" \
"confirm the run is really progressing"
return
fi
if [ "$wclass" = parked ]; then
# The run is stopped at a decision gate AND the worker is confirmed
# alive: the pane is idle because the crew asked a question and waited,
# which is the behavior its brief requires. Hold the ladder on the same
# terms as an active run - including the bounded recheck, so a gate
# nobody ever answers still cannot rot invisibly.
ladder_hold "$win" "$since_file" "$label" \
"the run is parked at a decision gate and this worker is still alive" \
"answer the gate or confirm the decision is still pending"
return
fi
n=$(( $(cat "$escalation_file" 2>/dev/null || echo 0) + 1 ))
Expand Down Expand Up @@ -726,6 +772,56 @@ codex_static_pane_upgrade() { # <window> <fallback-class>
printf '%s' "$fallback"
}

# parked_gate_liveness_class: decide what a `parked` absorb class (crew_absorb_class
# in bin/fm-classify-lib.sh - the authoritative run-step reports the pipeline stopped
# at a decision gate) is worth on the WEDGE-ESCALATION path, by corroborating it with
# a signal that does not read interface text: the harness-agent PROCESS.
#
# Scoped to the ladder on purpose: a gate only ever holds an escalation on a stale
# pane firstmate has ALREADY been woken for. The first sighting of that pane still
# surfaces at once, whatever the run-step says, because nothing has told anyone
# about the gate yet.
#
# The gate answers the wedge question outright for the crew that is still there. A
# worker parked at an ask-user finding is idle because it did exactly what its brief
# requires - ask, then stop. The 2026-08-08 evidence: two such workers escalated as
# `idle 254s, possible wedge, escalation 1` while fm-crew-state.sh, run against the
# same tasks at the same moment, returned `state: parked - source: run-step - parked
# at review: 3 finding(s) (ask-user: captain decision)`. The fleet already knew.
#
# But `parked` alone is NOT that answer, and absorbing on it would be the failure
# this whole absorb path exists to avoid. A worker that crashed one second after
# printing its gate prompt leaves the run parked in exactly the same way, forever:
# no-mistakes has no idea its agent is gone, so the run-step reading is IDENTICAL
# for a healthy gate and a dead one. Only `alive` - a confirmed agent process, the
# one piece of evidence that changes when the worker dies - licenses the absorb:
# alive -> parked (hold the ladder on the long bounded cadence)
# dead -> none (the caller's ordinary escalation path: a run parked with
# nobody left to answer it is a real failure, and it must
# reach firstmate on the unchanged timings, never be
# swallowed by the parked case)
# unknown -> none (fm_backend_agent_alive's contract: never license an action
# from unknown. Today that covers pi, whose launcher execs
# into a generic `node`, and every backend past tmux/herdr -
# those crews keep exactly today's behavior)
# Secondmates are excluded like the codex backstop excludes them: they never drive
# a run, so a `parked` verdict cannot be theirs to begin with.
#
# COST: this adds no crew-state read anywhere. The wedge escalation already had the
# fm-crew-state.sh verdict in hand, and the only new work is one liveness probe on
# the branch where that verdict is `parked`, once per escalation window. Nothing
# here runs per poll per task, which is the budget the watcher's whole absorb design
# is built around.
parked_gate_liveness_class() { # <window>
local win=$1 alive
[ "$(window_kind "$win")" != secondmate ] || { printf 'none'; return; }
alive=$(fm_backend_agent_alive "$(window_backend "$win")" "$win" 2>/dev/null) || alive=unknown
case "$alive" in
alive) printf 'parked' ;;
*) printf 'none' ;;
esac
}

pause_state_class() { # <window> <task>
local win=$1 task=$2 key last recheck_file class agent_alive
key=${win//:/_}
Expand All @@ -735,7 +831,24 @@ pause_state_class() { # <window> <task>
recheck_file="$STATE/.paused-rechecked-$key"
if ! status_is_paused_or_captain_held "$last"; then
rm -f "$recheck_file"
codex_static_pane_upgrade "$win" "$(crew_absorb_class "$task")"
class=$(crew_absorb_class "$task")
# A gate reading never absorbs a FIRST sighting, for ANY harness: at first
# sight nothing has relayed the gate to firstmate yet (this path is reached
# precisely when the crew's last status line is not captain-relevant), so
# swallowing it would leave the decision waiting on the long bounded cadence
# with nobody told. The gate only ever holds the wedge LADDER, once the pane
# is a known stale firstmate has already seen (wedge_timer_check).
# It short-circuits BEFORE codex_static_pane_upgrade rather than being folded
# into its `none` fallback: that backstop answers "no run-step says anything,
# is the process still there?" for a codex worker mid-turn on a static pane,
# and a run-step that authoritatively reports the run STOPPED at a gate is not
# that question. Routing the gate through it would hand codex alone the
# first-sight absorb every other harness just lost.
if [ "$class" = parked ]; then
printf 'none'
return
fi
codex_static_pane_upgrade "$win" "$class"
return
fi
if [ -e "$STATE/.paused-$key" ] && [ "$(age_of "$recheck_file")" -lt "$STALE_ESCALATE_SECS" ]; then
Expand All @@ -751,6 +864,12 @@ pause_state_class() { # <window> <task>
return
fi
class=$(crew_absorb_class "$task")
# Below this line the crew has DECLARED a pause or a verified captain hold, and
# the branch's own dead-agent rule already owns that idle pane. The run-step gate
# reading adds nothing here and its liveness corroboration would collide with the
# declaration's, so `parked` keeps the exact `none` handling it had before the
# gate class existed; only the undeclared path above consults it.
[ "$class" = parked ] && class=none
if [ "$class" = working ]; then
rm -f "$recheck_file"
printf 'working'
Expand Down
Loading
Loading