fix(insights): alerts must describe the present — stop re-firing about healed problems (v0.297.1) - #546
Merged
Merged
Conversation
…t healed problems (v0.297.1) Both repeat-offender alert branches were monotonic counters over a fixed window, so once a condition healed the count stayed above the threshold and the alert re-fired every 3-day cooldown until the rows aged out. - agent-crash: counted crashes over 30 days. On live instapods the consolidator crashed 9x during the TASK_B64 tmux-overflow bug (fixed in v0.265.2), then ran 12/12 green — and the alert still fired on 07-28, 07-31 and 08-03, telling the owner to scope a healthy agent's tasks smaller. Now reads a 7-day crashedRecent count and stands down after 3 clean runs since the last crash (runsSinceCrash). - friction: counted rejections over ALL TIME — no window, so it could never stop. stripe.refund (30 rejections, last one 2026-07-04) had alerted 7x in 30 days and would have gone forever. Now windowed to the same 30 days as the scorecard, which also clears stale entries off the Insights Friction card. - agent-low is left alone on purpose: it's a rate, so successes dilute it and it self-heals. The 30d `crashed` total stays on the scorecard — that surface is history; only the alert claims the present tense. New scripts/alert-staleness-test.cjs (wired into npm run test:governance) pins both directions: ongoing loops still alert, healed ones go quiet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The report
Getting this repeatedly. It's a false positive, and the advice is actively wrong.
What the live data says
All 9 consolidator crashes are in one dead 5-day window:
That gap is v0.265.2 (#463) shipping — the
TASK_B64tmux command-line overflow. The crash loop was diagnosed and fixed 10 days ago. Butagent-crash:consolidatorfired on 07-22, 07-25, 07-28, 07-31, 08-03 — every 3 days, exactly the cooldown — and the last three fired after the fix, against a 12/12-green agent. It would keep going until 2026-08-23.Same bug, worse, in the friction branch: the rejections query has no time window at all.
stripe.refund— 30 rejections, all between 2026-06-12 and 2026-07-04 — has alerted 7× in the last 30 days and would have alerted forever. The onlystripe.refundevents in the audit log for the past month are the alerts about it.Root cause
Both are monotonic counters over a fixed window being asked to answer a present-tense question. A crash count only goes up; once a loop is fixed the total stays over the threshold for the rest of the window. A scorecard can be history — an alert cannot, because it's an interrupt that costs a human attention and sends them to "fix" something healthy.
The fix
agent-crashreads a new 7-daycrashedRecentinstead of the 30d total, and stands down once the agent logs 3 clean runs since its last crash (runsSinceCrash). A fixed crash loop goes quiet immediately rather than ~4 weeks later. The 30dcrashedtotal stays on the scorecard — that surface is history.frictionis windowed to the same 30 days as the scorecard. This also drops stale rows off the Insights Friction card, which was showing an all-time count next to a 30-day scorecard.agent-lowdeliberately unchanged — it's a rate, so incoming successes dilute it and it self-heals. No guard needed.Verification
A/B on a
VACUUM INTOsnapshot of the live instapods DB (isolatedAGENT_OS_HOME), same build otherwise:success-dropsurvives in both — it's week-over-week, already recency-correct. Proof this narrows the noise rather than muting the channel.Negative control — same DB with the crashes shifted into the last 3 days: alert fires again, now reading
9 … in the last 7 days. Boundary: 2 clean runs since the crash → still fires; 3 → stands down.New
scripts/alert-staleness-test.cjs(12 assertions, wired intonpm run test:governance) pins both directions.npm run typecheck·webbuild ·test:governance265/265 green.🤖 Generated with Claude Code