fix(bin): surface certsync health cannot-run states

[Freudator86]

Intent

Stop firstmate's certsync health check (bin/fm-watch.sh: certsync_health_reason) from reporting healthy when it cannot run at all. Reproduced the reported defect: this account has no root/sudo/docker-group by design, so docker compose exec fails with a permission denial (nonzero exit, empty stdout); the shared helper run_bounded discarded exit codes via '... 2>/dev/null || true', so that failure collapsed into the exact same silent no-wake outcome as a confirmed-healthy reading. Fix: run_bounded now returns the wrapped command's real exit status (its only other caller, a fire-and-forget git fetch in the Bridge-inbox loop, never reads that status, so it is provably unaffected - proven with a new dedicated test plus the existing Bridge-inbox test suite passing unchanged). certsync_health_reason now treats every inability to read certsync's status - missing docker/jq, a failed status command, empty output, invalid JSON, a missing healthy boolean - as its own distinct 'cannot run' check wake, reusing the existing hourly resurface-dedup so it neither goes silent nor spams. Only a missing project/compose file (certsync not deployed on this host) stays legitimately quiet. Deliberately chose 'report as its own distinct wake' over inventing a new alarm channel or refusing outright, since the check already rides the watcher's existing heartbeat/check-wake infrastructure and a human should decide what to do about a docker permission gap that has a known bounded operator remediation. Did NOT fix the underlying docker permission denial - the account intentionally has no docker-group access and I was told not to propose adding it - instead documented a docker-socket-proxy runbook for the privileged operator who owns that fix (rejecting a same-shape differently-named raw-socket group as not actually narrower). Updated the existing test that had pinned the OLD buggy silent-absorb behavior (test_heartbeat_certsync_unknown_absorbed) to assert the new distinct cannot-run wake, and added a permission-denied regression test reproducing this exact defect end-to-end through the real watcher subprocess, plus a focused run_bounded contract test. Also updated docs/configuration.md to describe the corrected contract.

What Changed

• Captain, certsync heartbeat checks now surface cannot run check wakes for missing tools, failed status commands, empty output, invalid JSON, or missing healthy data instead of treating unreadable status as quiet.
• run_bounded now preserves the wrapped command's exit status while still discarding stderr, so callers can distinguish command failure from empty successful output.
• Adds regression coverage for run_bounded, certsync cannot-run watcher triage, and the docker permission-denied failure path, with docs updated for the corrected health-check contract.

Risk Assessment

✅ Low: Captain, the change is focused, preserves the only shared helper call site’s behavior, and makes certsync unreadable states fail visible without broad watcher changes.

Testing

Captain, I inspected the certsync/run_bounded diff, ran the focused helper contract test, the full watcher triage suite, and the Bridge inbox suite; then I manually reproduced the permission-denied docker path through the real watcher and drained queue. One initial manual transcript command was rejected by the local guard due to an rm -rf cleanup shape, then rerun successfully without deletion. The working tree was clean afterward, and no UI screenshot was needed because this is watcher/CLI behavior.

Evidence: Manual certsync permission-denied watcher transcript

watcher stdout: check: certsync health: cannot run: status command failed (exit 1) drained wake queue: 1785884768 1 check certsync-health check: certsync health: cannot run: status command failed (exit 1)

manual certsync permission-denied watcher check
manual directory: /tmp/no-mistakes-evidence/01KZ7G1T44XT852X50BFSCZFJT/manual-certsync-permission-denied.yj4GB2
watcher stdout:
check: certsync health: cannot run: status command failed (exit 1)
drained wake queue:
1785884768	1	check	certsync-health	check: certsync health: cannot run: status command failed (exit 1)

Evidence: Watcher triage regression test log

ok - heartbeat surfaces a docker-permission-denied certsync status instead of silently reporting healthy

ok - signal_reason_is_actionable: benign absorbed, captain verbs and coalesced batches surfaced
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - scan_captain_relevant_statuses lists only captain-relevant statuses
ok - classifier primitives: keyed decisions and activity phases, captain relevance, window-to-task, and overrides
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - status_is_paused: only the leading paused verb matches, and paused is not captain-relevant
ok - crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a two-file single crewmate turn enqueues exactly one wake while both markers advance
ok - away mode collapses a two-file crewmate turn to one wake while both markers advance
ok - two crewmates signalling in one poll still produce one wake each (dedup never crosses tasks)
ok - a turn-end marker moving alone still keys its own wake
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a relayed terminal task absorbs parked pane churn and re-surfaces on the bounded cadence
ok - a real status write wakes immediately and clears parked tracking
ok - a metadata change clears parked tracking before stale classification
ok - mark-parked wrapper: creates the marker for a recorded window, refuses an unrecognized one, never engages the watcher lock
ok - mark-parked wrapper: refuses a kind=secondmate window, leaving the pause-tracking path untouched
ok - a stale terminal-looking status is absorbed while its run works, holds the ladder, and escalates once the run stops
ok - provably-working non-terminal stale is absorbed, holds the ladder while its run is active, and escalates once the run stops
ok - a healthy static-pane codex worker with an alive process is absorbed (agent-liveness backstop), and the wedge timer still arms
ok - a codex pane whose process is dead surfaces immediately (the backstop never masks a crash)
ok - the agent-liveness absorb is scoped to codex: a claude worker with an alive process still surfaces immediately
ok - a follow-on stale repeating an already-surfaced terminal line is absorbed, and a real wedge behind it still escalates
ok - a stale whose captain-relevant line has not been surfaced yet still wakes firstmate at once
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - the ladder holds for a provably-working crew under a stripped environment, and an unreadable read is reported instead
ok - a ladder hold that lasts a full recheck window surfaces one bounded recheck without climbing the ladder
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - exited declared-pause and captain-held panes use bounded pause cadence while a live decision gate still surfaces once
ok - a declared paused secondmate re-surfaces on the bounded normal-mode cadence
ok - a non-paused secondmate retains normal stale suppression
ok - a resumed secondmate clears pause and stale tracking before stale exemption
ok - unchanged stale hashes reclassify when a crew enters or leaves pause
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working keeps its wedge timer and holds the ladder while the run is active
ok - a ladder hold on the paused-then-working path surfaces one bounded recheck without climbing the ladder
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - triage log capping handles wc byte counts with leading spaces
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - heartbeat absorbs healthy certsync status
ok - heartbeat surfaces confirmed unhealthy certsync through the check wake path
ok - afk heartbeat surfaces confirmed unhealthy certsync through the check wake path
ok - heartbeat surfaces an unreadable certsync status as its own cannot-run check wake, distinct from healthy
ok - heartbeat surfaces a docker-permission-denied certsync status instead of silently reporting healthy
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)
ok - AFK changed paused panes hand off plain stale identities for daemon-owned pause triage

Evidence: run_bounded helper contract log

ok - a caller that never reads run_bounded's exit status (the Bridge fetch call site's shape) is unaffected by the real exit code now propagating

ok - run_bounded returns the wrapped command's real exit status instead of always reporting success
ok - run_bounded still reports success for a command that actually succeeds
ok - run_bounded still discards the wrapped command's stderr (only its exit status changed)
ok - a caller that never reads run_bounded's exit status (the Bridge fetch call site's shape) is unaffected by the real exit code now propagating
# fm-watch-run-bounded.test.sh: all assertions passed

Evidence: Bridge inbox suite log

ok - origin ack clears the check without mutating a stale local working tree

ok - Bridge vessel resolution prefers a non-empty env value and falls back to per-home config
ok - a space-separated FM_BRIDGE_VESSEL resolves into an ordered vessel list, with BRIDGE_VESSEL as the first
ok - unconfigured home performs no Bridge scan and emits no wake
ok - Bridge inbox wakes once per pending signature, clears on ack, and re-fires on re-delivery
ok - each configured vessel surfaces its own pending mail independently of the others
ok - empty Bridge inbox is absorbed silently
ok - high-priority Bridge traffic tightens only its poll interval
ok - the shared Bridge cadence tightens when any watched vessel is high or immediate priority, not only the primary
ok - unchanged Bridge inbox reuses the cached priority; new arrivals trigger a rescan
ok - an in-place envelope edit under an unchanged filename invalidates the cached priority
ok - missing Bridge inbox directory short-circuits without spawning a scan
ok - repeated unchanged loop checks do not keep spawning Bridge scans within the urgent window
ok - origin ack clears the check without mutating a stale local working tree

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• bin/fm-session-start.sh
• git diff --stat deb3593458529e439b11d7d0a183be12f3bb3750..37d807208b32eed3116acaa55cadccd3d6cf24d0
• git diff --name-only deb3593458529e439b11d7d0a183be12f3bb3750..37d807208b32eed3116acaa55cadccd3d6cf24d0
• rg -n "certsync_health_reason|run_bounded|test_heartbeat_certsync|permission" bin test tests docs
• tests/fm-watch-run-bounded.test.sh | tee /tmp/no-mistakes-evidence/01KZ7G1T44XT852X50BFSCZFJT/fm-watch-run-bounded.log
• tests/fm-watch-triage.test.sh | tee /tmp/no-mistakes-evidence/01KZ7G1T44XT852X50BFSCZFJT/fm-watch-triage.log
• tests/fm-watch-bridge-inbox.test.sh | tee /tmp/no-mistakes-evidence/01KZ7G1T44XT852X50BFSCZFJT/fm-watch-bridge-inbox.log
• Manual watcher subprocess check with a fake docker compose permission denial, followed by bin/fm-wake-drain.sh, recorded at /tmp/no-mistakes-evidence/01KZ7G1T44XT852X50BFSCZFJT/manual-certsync-permission-denied.txt
• git status --short

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.