Skip to content

AFK away-mode: sub-supervisor never delivers escalations to the primary pane (composer reads 'pending'), wedges for hours — recurred across 3 sessions #21

Description

@Quidge

Summary

Across the last three away-mode (/afk) sessions — 2026-08-05, -06, -07 — the sub-supervisor daemon (bin/fm-supervise-daemon.sh) could not inject any escalation into the primary/captain pane. Every injection attempt was deferred, the daemon logged repeated wedge alarms, and no escalation was delivered live. In each case the buffered escalations were recovered only later, via the fm-afk-return.sh catch-up drain when the captain returned. Data integrity held — nothing was lost — but the captain got zero live escalations while away.

This ticket records what happened + trace data only. Root cause and fix are intentionally left open for later triage.

Impact

  • While away, the captain received no live escalations (review-ready PRs, done/blocked events); they surfaced only on return.
  • Undelivered spans (peak per session): Aug 5 ≈ 6083s (~1.7h), Aug 6 ≈ 27241s (~7.5h), Aug 7 ≈ 8769s (~2.4h).
  • On Linux there is no OS-level wedge alert channel and config/wedge-alarm is unset, so the only wedge signal was the durable state/.subsuper-inject-wedged marker — which an away captain never sees.

Affected sessions (3 daemon starts, identical symptom)

Daemon start (UTC) PID inject-deferrals that date peak undelivered
2026-08-05T22:13:43 20188 438 6083s
2026-08-06T08:49:24 2970957 2010 27241s
2026-08-07T06:08:57 954385 622 8769s

All started with: target=%0 (primary pane = window 0 / fm-MAIN), backend=tmux, afk=on, inject_skip='heartbeat', stale_escalate=240s, batch=90s.

Per-session deferral windows: Aug5 22:15:22→23:59:53 · Aug6 00:00:08→16:27:03 · Aug7 06:10:36→08:38:45.

Dominant symptom (from the daemon log)

Injection deferrals by reason (whole 3431-line log):

  • inject deferred: supervisor composer not confirmed-empty (state=pending: pending input, dead-shell prompt, or unreadable pane)3063
  • inject deferred: supervisor pane busy (agent mid-turn)7

So injection was almost always blocked by the composer-state guard reading the primary pane as state=pending, not by the busy guard. Representative (Aug 7), repeating every ~15s:

[2026-08-07T06:10:36+0000] inject deferred: supervisor composer not confirmed-empty (state=pending: pending input, dead-shell prompt, or unreadable pane)
[2026-08-07T06:10:51+0000] inject deferred: supervisor composer not confirmed-empty (state=pending: pending input, dead-shell prompt, or unreadable pane)

Delivery outcome

  • No inject delivered / confirmed / submit confirmed line appears anywhere in the log. The only escalate: lines (4 total across the 3 sessions) are the daemon deciding to buffer an escalation. Combined with 3070 deferrals + 288 wedge-alarm ERROR lines, it appears no injection ever landed live in any of the 3 sessions; all were recovered by the return catch-up drain.
  • The 4 buffered escalate events (what was waiting to deliver): firstmate-dev status (Aug 5, issue-triage), oikos-dev + oikos-skill-adoption (Aug 6), a stale: firstmate (Aug 7), and the firstmate-dev upstream-sync PR (Aug 7).

Wedge-alarm behavior

Every ~5 min while wedged the daemon logged (example, worst case Aug 6):

[2026-08-06T16:23:27+0000] ERROR: away-mode escalation undelivered 27241s; inject could not confirm a submit (supervisor pane busy or wedged). Buffer + wake-queue preserved; alarm marker written.
[2026-08-06T16:23:27+0000] wedge alarm: no OS-level alert channel on Linux; durable marker /home/lirum/firstmate/state/.subsuper-inject-wedged is the only signal - set config/wedge-alarm (e.g. a command: directive)

Return / shutdown anomaly (Aug 7)

On tonight's return, the first fm-afk-return.sh reported away-mode daemon did not exit after SIGTERM; preserving lifecycle state / away-mode shutdown failed; lifecycle state preserved for retry. A retry of bin/fm-afk-launch.sh stop then succeeded (away mode stopped; daemon terminal torn down and .afk cleared). Daemon log's last line: [2026-08-07T08:38:58+0000] daemon shutting down; background job exited 0. (Whether the same SIGTERM behavior occurred on the Aug 5/6 returns was not checked.)

Environment

  • Primary harness claude (Opus 4.8, claude-opus-4-8[1m]); backend tmux; box myrmex (Linux, Ubuntu).
  • Supervisor pane = window 0 (fm-MAIN) — the captain's own interactive claude session (NOT firstmate-launched).
  • config/wedge-alarm unset; secondmate/crew harness claude.
  • Native start path used: fm-afk-launch.sh start-native + backgrounded fm-afk-start.sh (per the afk skill's claude native-background path).

Trace artifacts preserved on-box (gitignored data/)

  • data/afk-wedge-2026-08-07/supervise-daemon.log — full daemon log, 543 KB / 3431 lines, spanning 2026-08-05T22:13 → 2026-08-07T08:38 (all 3 sessions).
  • data/afk-wedge-2026-08-07/evidence-summary.txt — extracted counts + excerpts.
  • Preserved because the live daemon log (state/.supervise-daemon.log) is session-scoped and cleared on a fresh /afk entry.

Context that MAY help triage (pointers, not a diagnosis)

  • The composer-state guard + ghost/placeholder handling for tmux live in bin/fm-composer-lib.sh and bin/fm-tmux-lib.sh; the injection-safety contract is documented in the afk skill sections "Busy-guard and composer guard" and "Submit model".
  • The supervisor pane is the captain's own claude session. Per harness-adapters (claude section), firstmate launches CREW/secondmate claude panes with CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false to suppress predicted-prompt ghost text, and the shared fm_composer_strip_ghost extractor is the fallback for panes that flag cannot reach. Noted only as a relevant code path to examine, given the guard consistently read state=pending.

What was NOT determined (left open per captain)

  • Why the composer guard read the primary pane as pending (genuine unsubmitted text vs ghost/placeholder vs unreadable) — not investigated.
  • Whether the max-defer flush / stale_escalate path ran and why it also couldn't submit.
  • The cause of the SIGTERM-didn't-stop-on-first-try behavior.
  • Whether this reproduces with a non-claude primary or a different backend.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions