Skip to content

fix(insights): alerts must describe the present — stop re-firing about healed problems (v0.297.1) - #546

Merged
vikasprogrammer merged 1 commit into
mainfrom
feat/alert-staleness
Aug 3, 2026
Merged

fix(insights): alerts must describe the present — stop re-firing about healed problems (v0.297.1)#546
vikasprogrammer merged 1 commit into
mainfrom
feat/alert-staleness

Conversation

@vikasprogrammer

Copy link
Copy Markdown
Owner

The report

9 of consolidator's runs crashed in the last 30 days — the process died mid-run … Scope its tasks smaller, or give it more headroom.

Getting this repeatedly. It's a false positive, and the advice is actively wrong.

What the live data says

All 9 consolidator crashes are in one dead 5-day window:

Crashes 2026-07-20 → 2026-07-24 16:35 (9 runs, each dead in ~10s)
First success 2026-07-24 16:42 — 7 minutes later
Since then 12 runs, 12 done, 0 crashed

That gap is v0.265.2 (#463) shipping — the TASK_B64 tmux command-line overflow. The crash loop was diagnosed and fixed 10 days ago. But agent-crash:consolidator fired on 07-22, 07-25, 07-28, 07-31, 08-03 — every 3 days, exactly the cooldown — and the last three fired after the fix, against a 12/12-green agent. It would keep going until 2026-08-23.

Same bug, worse, in the friction branch: the rejections query has no time window at all.

SELECT capability, count(*) FROM approvals WHERE status = 'rejected' GROUP BY capability

stripe.refund — 30 rejections, all between 2026-06-12 and 2026-07-04 — has alerted 7× in the last 30 days and would have alerted forever. The only stripe.refund events in the audit log for the past month are the alerts about it.

Root cause

Both are monotonic counters over a fixed window being asked to answer a present-tense question. A crash count only goes up; once a loop is fixed the total stays over the threshold for the rest of the window. A scorecard can be history — an alert cannot, because it's an interrupt that costs a human attention and sends them to "fix" something healthy.

The fix

  • agent-crash reads a new 7-day crashedRecent instead of the 30d total, and stands down once the agent logs 3 clean runs since its last crash (runsSinceCrash). A fixed crash loop goes quiet immediately rather than ~4 weeks later. The 30d crashed total stays on the scorecard — that surface is history.
  • friction is windowed to the same 30 days as the scorecard. This also drops stale rows off the Insights Friction card, which was showing an all-time count next to a 30-day scorecard.
  • agent-low deliberately unchanged — it's a rate, so incoming successes dilute it and it self-heals. No guard needed.

Verification

A/B on a VACUUM INTO snapshot of the live instapods DB (isolated AGENT_OS_HOME), same build otherwise:

CONTROL (main)                          FIXED
─────────────────────────────────       ─────────────────────────────────
[high]   success-drop                   [high] success-drop
[high]   agent-crash:consolidator   →   (gone — crashedRecent=0, sinceCrash=11)
[medium] friction:stripe.refund     →   (gone — rejections=[])

success-drop survives in both — it's week-over-week, already recency-correct. Proof this narrows the noise rather than muting the channel.

Negative control — same DB with the crashes shifted into the last 3 days: alert fires again, now reading 9 … in the last 7 days. Boundary: 2 clean runs since the crash → still fires; 3 → stands down.

New scripts/alert-staleness-test.cjs (12 assertions, wired into npm run test:governance) pins both directions.

npm run typecheck · web build · test:governance 265/265 green.

🤖 Generated with Claude Code

…t healed problems (v0.297.1)

Both repeat-offender alert branches were monotonic counters over a fixed window, so
once a condition healed the count stayed above the threshold and the alert re-fired
every 3-day cooldown until the rows aged out.

- agent-crash: counted crashes over 30 days. On live instapods the consolidator
  crashed 9x during the TASK_B64 tmux-overflow bug (fixed in v0.265.2), then ran
  12/12 green — and the alert still fired on 07-28, 07-31 and 08-03, telling the
  owner to scope a healthy agent's tasks smaller. Now reads a 7-day crashedRecent
  count and stands down after 3 clean runs since the last crash (runsSinceCrash).
- friction: counted rejections over ALL TIME — no window, so it could never stop.
  stripe.refund (30 rejections, last one 2026-07-04) had alerted 7x in 30 days and
  would have gone forever. Now windowed to the same 30 days as the scorecard, which
  also clears stale entries off the Insights Friction card.
- agent-low is left alone on purpose: it's a rate, so successes dilute it and it
  self-heals.

The 30d `crashed` total stays on the scorecard — that surface is history; only the
alert claims the present tense. New scripts/alert-staleness-test.cjs (wired into
npm run test:governance) pins both directions: ongoing loops still alert, healed
ones go quiet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vikasprogrammer
vikasprogrammer merged commit 917c67a into main Aug 3, 2026
1 check passed
@vikasprogrammer
vikasprogrammer deleted the feat/alert-staleness branch August 3, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant