Skip to content

fix(monitors): surface persistent poll failures instead of silent catch - #332

Merged
dngioidev merged 1 commit into
mainfrom
fix/318-monitor-error-surfacing
Aug 1, 2026
Merged

fix(monitors): surface persistent poll failures instead of silent catch#332
dngioidev merged 1 commit into
mainfrom
fix/318-monitor-error-surfacing

Conversation

@dngioidev

Copy link
Copy Markdown
Owner

Closes #318 (parent epic #183)

Problem

Both autopilot monitors wrapped each poll in catch { /* transient */ }, swallowing all errors forever. A persistent auth/network failure silenced the CI monitor permanently (the loop never learns CI state); a persistent fs error meant resolved decisions never surfaced, so parked/blocked tickets never unblock — a silent stall of the escalation-resume path.

Fix (thin, poll cadence preserved)

  • New shared plugin/scripts/monitors/poll-guard.mjs — a pure consecutive-failure guard. Reset on any good poll; emit exactly one <monitor> error: <reason> (N consecutive polls) line once the streak reaches FAILURE_THRESHOLD (3); then throttle (REEMIT_EVERY = 10) so a lasting outage emits once per episode, never per poll. Keeps polling throughout.
  • ci-watch.mjs: poll now returns ok/reason; distinguishes benign "no PR yet" (isNoPr) from a real gh failure. Tick applies the guard.
  • decisions-watch.mjs: readDecisions rethrows non-ENOENT fs errors (ENOENT/missing dir stays benign); poll returns { lines, ok, reason }. Tick applies the guard.
  • Healthy/quiet/transition behavior and the emit contract are unchanged.
  • SKILL.md monitor section documents the new error line.

Acceptance criteria

  • AC.1 — a monitor failing repeatedly (N consecutive polls) surfaces a one-line error to the loop instead of staying silent. Verified: AC-318.1 guard test asserts exactly one line at the threshold and that the counter keeps advancing (polling continues); AC-318.1 decisions test forces a real fs error (ENOTDIR) → ok:false.
  • AC.2 — transient single failures do not spam; test covers transient-vs-persistent. Verified: AC-318.2 tests — a single/below-threshold failure surfaces nothing, and a success resets the counter so occasional blips never accumulate; ci poll marks a real gh failure ok:false but stays ok on no-PR.

Verification

  • pnpm verify: 611/611 pass (monitors file 11/11, up from 4 — 7 new).
  • No live-run impact (installed cache is separate from repo source).

The two autopilot monitors wrapped every poll in `catch { /* transient */ }`,
so a standing failure silenced the watcher forever: a persistent auth/network
error left the CI monitor mute (the loop never learned CI state), and a real
fs error left resolved decisions un-surfaced (parked tickets never unblock).

Add a shared consecutive-failure guard (poll-guard.mjs): reset on any good
poll, emit exactly one concise `<monitor> error: <reason> (N consecutive polls)`
line once the streak hits a threshold (N=3), then throttle (re-emit at most
once per 10 further failures) so a lasting outage never spams. A single/
transient failure stays silent. ci-watch distinguishes benign "no PR yet" from
a real gh failure; decisions-watch distinguishes a missing dir from a real fs
error. Poll cadence and all healthy/quiet/transition behavior are unchanged.

Closes #318

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SATRHKa6mDHDuirhP6QuwL
@dngioidev
dngioidev merged commit 1dd5d4d into main Aug 1, 2026
10 of 11 checks passed
@dngioidev
dngioidev deleted the fix/318-monitor-error-surfacing branch August 1, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autopilot monitors: surface persistent errors instead of silent catch

1 participant