Fix four drill-exposed detection gaps: verify-deploy blind window, dormant carve-out, change correlation, error grouping#307
Merged
Conversation
… to it
Phase 3 used to query Loki for {container="..."} errors over "the last 2
minutes" immediately after ArgoCD reported Health=Healthy — while the
rollout could still be finishing and the OLD, still-serving pod was what
the broad container selector actually sampled. The 2026-07-26 drill (a
background loop throwing every 15s + a throwing Telegram update handler)
shipped, verify-deploy.sh PASSED (run 30259908859), and the bot was
completely non-functional: the first drill error appeared at 11:02:44,
strictly after Phase 3's query had already run and passed.
Fix:
- Capture READY_AT the moment ArgoCD confirms Healthy (both the in-loop and
end-of-grace-period paths).
- Sleep a fixed LOKI_SETTLE_DELAY (default 45s = 3x the drill's 15s
failing-loop period) after readiness, so at least two iterations of a
periodic background failure have time to fire and propagate through the
scrape+ship pipeline (typical 10-15s scrape interval + Loki ingestion lag)
before Phase 3 samples.
- Resolve the NEW pod's name(s) via the ArgoCD resource-tree endpoint
(ResourceNode.images, same mechanism sre.md already uses to tell old/new
pods apart) and scope the Loki query to them with pod=~"...". Falls back
to the broad container selector (logged + summarized) if resource-tree is
unreachable or no pod matches the expected image tag, rather than failing
the whole check.
- Start the Phase 3 Loki window at READY_AT instead of "2 minutes ago from
whenever Phase 3 happens to run".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
The 2026-07-26 drill exposed the monitor agent suppressing a genuine error burst on alita because it read "traffic_class: dormant disables rules 3/4" as applying to EVERY series, including log_errors_24h itself: `current=13 median_28d=0 z_score_28d=33.59 emerged_from_zero=true emerged_from_zero_significant=true` was handed to it and dropped, reasoning "rules 3 and 4 ... are disabled for this bot." The carve-out exists only to stop alita's zero-inflated TRAFFIC/VOLUME series from firing constantly; it must never apply to error/failure series. - scripts/gather/baseline.sh: `_stats_jq` now emits `dormant_exempt` per series (false for log_errors_24h/log_warnings_24h — rules always apply; true for everything else), mirroring the existing `informational_only` pattern so the distinction is mechanical rather than prose the agent has to remember. - .github/prompts/monitor.md: the "Dormant carve-out" section and rule 4's inline exception note now gate on `dormant_exempt` explicitly, quote the drill's own evidence as the cautionary example, and state plainly that rules 1/2/5 already apply unconditionally regardless of traffic_class. Fixture-tested (synthetic 28-day alita history, log_errors_24h flat at 0 until a current=13 spike, message_log_24h zero-inflated as alita's real pattern is): log_errors_24h comes back `dormant_exempt: false, emerged_from_zero: true, emerged_from_zero_significant: true`; message_log_24h comes back `dormant_exempt: true`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
…ding The 2026-07-26 drill agent wrote: "Classification: attributed — the increased error count is plausibly explained by the recent intentional DRILL changes... that makes the observed error volume expected" and suppressed a genuine, bot-breaking error burst. A deploy explaining an error burst does not make it benign — it makes it a bad deploy. Most real outages ARE deploy-correlated, so unscoped `attributed` suppression throws away exactly the incidents that matter most. Restricts `attributed` (suppress, do not file) to traffic/volume findings only (rule 3/4 candidates on a series with `dormant_exempt: true`). For error-rate findings (rule 1, rule 2, or rule 3/4 on log_errors_24h / log_warnings_24h), a correlated deploy now RAISES severity: classify as `correlated — suspected bad deploy`, file priority-high, name the suspected commit. Two worked examples included verbatim: the existing vahter PR #213 log-volume case (volume, correctly suppressed) and the 2026-07-26 drill itself (error-rate, must be filed, not suppressed) quoting the actual wrong classification from that run. Also updated the "Change correlation" field in the finding-issue template so a filed error-rate finding never says "attributed." Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
…dence
The 2026-07-26 monitor run reported rules 1 (new failure mode) and 2
(error-group z-score) UNEVALUABLE: "per-group identifiers and z-scores
were not present in the provided Loki grouping output (only counts-only
lines were included)". The drill's brand-new Program.DrillRuntimeFailureLoop
SourceContext - a guaranteed rule-1 hit, unseen in 28 days - was invisible
behind a bare "N occurrence(s)" line.
- scripts/gather/runtime.sh: TOP_ERRORS_GROUPED (was TOP_ERRORS_COUNT) now
groups by SourceContext + 80-char message prefix and renders each group
as its SourceContext, count, prefix, AND one verbatim raw sample line -
this is what actually reaches the monitor agent's evidence bundle (via
runtime.sh's markdown output in monitor.yml), so this is the load-bearing
fix.
- scripts/gather/baseline.sh: cmd_gather's error_groups is now an ARRAY of
{source_context, message_prefix, count, sample} objects (was a bare
key->count map with no way to recover SourceContext or a sample line
from a key string), sorted by count descending. Also updated the empty
fallback ({} -> []) and header docstring to match the new shape.
- Both jq programs' message-field fallback now checks CLEF's @m first
($parsed["@m"] // .RenderedMessage // .message // .msg // $raw) - the
drill's binding facts confirmed the real Loki payload is CLEF-shaped
({"@t":...,"@l":"Error","@m":"...","SourceContext":"..."}), and the
prior fallback chain never matched it, so message_prefix would have
silently degraded to the raw JSON blob.
- query_loki_patterns still 404s on this Loki (per the brief) - grouping
stays hand-built; "| json | level=~\"Error|Fatal\"" queries are unchanged
and already verified working.
Fixture-tested against a canned Loki query_range response built from the
real CLEF shape (two SourceContexts: Program.DrillRuntimeFailureLoop x3,
AlitaBot.Bot.UpdateHandler x1) - both the runtime.sh and baseline.sh jq
transforms correctly emit per-group SourceContext, count, and a verbatim
sample.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
…ion rules Documentation-only sync of section 6.1 (monitor role spec) with the two prompt/mechanical fixes made in this branch (dormant carve-out scoping, error-rate vs traffic/volume change correlation) and the 2026-07-26 drill postmortem that motivated both, so the design doc and .github/prompts/monitor.md do not drift back out of sync. No behavior change - this commit touches only the design document, not scripts/gather/baseline.sh or monitor.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
monitor.yml injects DORMANT_NOTE into the agent's prompt at runtime. It still read "disables volume-based rules entirely for this bot" — restating the exact rule that defect 2 removes, and contradicting the corrected monitor.md in the same prompt. That contradiction is how defect 2 happened: on 2026-07-27 the agent read the blanket wording and suppressed log_errors_24h at z_score 33.59, emerged from a 28-day median of 0, on a bot that was completely dead and not answering its owner. A prompt that contradicts its own rulebook resolves to whichever the model reads last. Fixing the rulebook without fixing the injected note would have left the bug reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU
Szer
added a commit
that referenced
this pull request
Jul 27, 2026
… for SRE-agent validation take 3 (#309) DELIBERATE, OWNER-AUTHORIZED SRE-AGENT VALIDATION DRILL. The owner explicitly authorized taking AlitaBot down again: its only chat is with them, and this re-run specifically validates the six detection-pipeline fixes from #307. Same failure SHAPE as drill 2 (#303, reverted in #304): a hosted BackgroundService (PendingUsageFlushLoop) ticking every 15s via PeriodicTimer, logging Error on every tick, traffic-independent so it fires within verify-deploy.sh's Loki window even though AlitaBot is dormant; plus a throwing Telegram update handler so real messages also fail. Both gated on `not TestMode`. The one thing that's DIFFERENT on purpose: drill 2's every log line contained the literal token SRE_DRILL_RUNTIME_FAILURE plus "deliberate"/"drill"/"must be reverted" — the monitor agent read those words in the log text itself and filed nothing while AlitaBot was completely dead. This drill's runtime log text and exception messages are deliberately disguised as an ordinary application bug instead: - PendingUsageFlushLoop raises via an unguarded `Seq.head` on an empty sequence (System.ArgumentException: "The input sequence was empty."), logged as "Usage flush tick failed while flushing pending LLM usage records" — reads like a forgot-to-check-empty-queue bug in a usage-flush tick, not a drill. - The webhook handler dereferences a null string (System.NullReferenceException: "Object reference not set to an instance of an object."), caught by the existing generic "Unhandled error in update handler for {UpdateId}" log — a completely ordinary NRE. No "drill"/"SRE_DRILL"/"intentional"/"revert" appears in any string literal, log message, or exception text — only in the F# source comments above each throw site, which state prominently and in capitals (for future readers of the diff/commit, not for Loki) that this is deliberate and must be reverted. Gate verified: prod bot_setting.TEST_MODE = 'false' (per #307's binding context); tests/AlitaBot.Tests sets TEST_MODE=true via both the container env var and a seeded bot_setting row (ContainerTestBase.fs lines 34, 87), so the drill code never executes under the hermetic suite. AlitaBot-only (src/AlitaBot/Program.fs), single commit, no BotInfra/other-bot/migration/ Helm/k8s/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 27, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the four detection defects exposed by today's (2026-07-26) deliberate production drill,
where a genuinely broken AlitaBot deploy (background loop throwing every 15s + a throwing
Telegram update handler, bot completely non-functional) was shipped GREEN by
verify-deploy.sh(run 30259908859) and then reported CLEAN by the monitor (run 30260458048).verify-deploy.shblind window (scripts/verify-deploy.sh) — Phase 3 now samples Lokionly after a settle delay past confirmed readiness, and scopes the query to the NEW pod's
name(s) resolved via the ArgoCD resource-tree endpoint (falls back to the broad selector,
logged, if that's unavailable). Settle delay defaults to 45s (3x the drill's 15s
failing-loop period).
scripts/gather/baseline.sh,.github/prompts/monitor.md) —baseline.shnow emits a mechanicaldormant_exemptflagper series (
falseforlog_errors_24h/log_warnings_24h,truefor traffic/volumeseries).
monitor.md's dormant carve-out now gates on this flag instead of a blanket"rules 3/4 disabled for dormant bots."
.github/prompts/monitor.md) —attributedcan now only suppress traffic/volume findings. Error-rate findings (new failure mode,
error-group z-score, log_errors_24h/log_warnings_24h ratio/emerged-from-zero) that
correlate with a deploy are now classified
correlated — suspected bad deployand filedpriority-highnaming the commit — never suppressed.SourceContext(scripts/gather/runtime.sh,scripts/gather/baseline.sh) — error-log grouping now emits per-group SourceContext,message prefix, count, AND one verbatim sample line (was counts-only), with the message
field fallback fixed to check CLEF's
@mfirst.Plus a documentation-only commit syncing
.github/AGENT-FLOWS-REDESIGN.md§6.1 with thecorrected dormant/change-correlation rules.
Four defect fixes as four separate, independently-revertible commits, plus one doc-sync
commit.
Test plan
bash -n+shellcheck -x(0.11.0) on every script touched — all clean, verbatim outputin the task report.
runtime.shandbaseline.sh) against acanned Loki
query_rangeresponse built from the real CLEF shape, with two distinctSourceContextvalues — verbatim output pasted in the task report.dormant_exemptflag against a synthetic 28-day alita history wherelog_errors_24hspikes from a flat-zero baseline (the drill scenario) whilemessage_log_24halso emerges from zero (alita's normal pattern) — confirmslog_errors_24h: dormant_exempt: falseandmessage_log_24h: dormant_exempt: true.dotnet test(not needed for this PR — nosrc//tests/changes) and no realTelegram/LLM tests (
ALITA_REAL_TESTS/COUPON_REAL_TESTSdeliberately not set, per repopolicy).
🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU