fix(monitors): surface persistent poll failures instead of silent catch - #332
Merged
Conversation
The two autopilot monitors wrapped every poll in `catch { /* transient */ }`,
so a standing failure silenced the watcher forever: a persistent auth/network
error left the CI monitor mute (the loop never learned CI state), and a real
fs error left resolved decisions un-surfaced (parked tickets never unblock).
Add a shared consecutive-failure guard (poll-guard.mjs): reset on any good
poll, emit exactly one concise `<monitor> error: <reason> (N consecutive polls)`
line once the streak hits a threshold (N=3), then throttle (re-emit at most
once per 10 further failures) so a lasting outage never spams. A single/
transient failure stays silent. ci-watch distinguishes benign "no PR yet" from
a real gh failure; decisions-watch distinguishes a missing dir from a real fs
error. Poll cadence and all healthy/quiet/transition behavior are unchanged.
Closes #318
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SATRHKa6mDHDuirhP6QuwL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #318 (parent epic #183)
Problem
Both autopilot monitors wrapped each poll in
catch { /* transient */ }, swallowing all errors forever. A persistent auth/network failure silenced the CI monitor permanently (the loop never learns CI state); a persistent fs error meant resolved decisions never surfaced, so parked/blocked tickets never unblock — a silent stall of the escalation-resume path.Fix (thin, poll cadence preserved)
plugin/scripts/monitors/poll-guard.mjs— a pure consecutive-failure guard. Reset on any good poll; emit exactly one<monitor> error: <reason> (N consecutive polls)line once the streak reachesFAILURE_THRESHOLD(3); then throttle (REEMIT_EVERY= 10) so a lasting outage emits once per episode, never per poll. Keeps polling throughout.ci-watch.mjs:pollnow returnsok/reason; distinguishes benign "no PR yet" (isNoPr) from a real gh failure. Tick applies the guard.decisions-watch.mjs:readDecisionsrethrows non-ENOENT fs errors (ENOENT/missing dir stays benign);pollreturns{ lines, ok, reason }. Tick applies the guard.Acceptance criteria
AC-318.1guard test asserts exactly one line at the threshold and that the counter keeps advancing (polling continues);AC-318.1decisions test forces a real fs error (ENOTDIR) →ok:false.AC-318.2tests — a single/below-threshold failure surfaces nothing, and a success resets the counter so occasional blips never accumulate; ci poll marks a real gh failureok:falsebut staysokon no-PR.Verification
pnpm verify: 611/611 pass (monitors file 11/11, up from 4 — 7 new).