Program.DrillRuntimeFailureLoop — background loop runtime failures (suspected bad deploy)

[github-actions]

Problem

Background loop runtime failures in AlitaBot's background loop (SRE-authorized validation drill) producing a large Error-level spike across the 24h window. This appears to be a new error mode observed in Loki and coincides with recent merges/deploys touching src/AlitaBot/.

Evidence

Program.DrillRuntimeFailureLoop (81x), verbatim sample:

{"@t":"2026-07-27T11:02:14.5275099Z","@m":"SRE_DRILL_RUNTIME_FAILURE: background loop tick failed (deliberate drill, revert immediately)","@i":"33960b10","@l":"Error","@x":"System.Exception: SRE_DRILL_RUNTIME_FAILURE: this is a deliberate, owner-authorized SRE-agent validation drill — AlitaBot's background loop is intentionally broken and must be reverted immediately after the drill completes.\n   at Program.ExecuteAsync@42.MoveNext() in /repo/src/AlitaBot/Program.fs:line 46","SourceContext":"Program.DrillRuntimeFailureLoop"}

Baseline comparison

log_errors_24h: current=83 median_7d=0 median_28d=0 ratio_vs_28d=null z_score_28d=36.744890922927645 history_days_28d=29

Change correlation

Suspected bad deploy: ArgoCD revision 60005602 (deployedAt 2026-07-27T11:01:56Z).

Recent merges to main touching src/AlitaBot/ in the preceding 72h (from change_context):

• 086e21e Revert "DRILL (revert after): AlitaBot intentional runtime failure for SRE-agent validation, take 2 (DRILL 2 (revert after): AlitaBot intentional runtime failure for SRE-agent validation #303)" (Revert drill 2: restore AlitaBot #304)
• 6100a4b DRILL (revert after): AlitaBot intentional runtime failure for SRE-agent validation, take 2 (DRILL 2 (revert after): AlitaBot intentional runtime failure for SRE-agent validation #303)
• a655e1f Revert "DRILL (revert after): AlitaBot intentional startup failure for SRE-agent validation (DRILL (revert after): AlitaBot intentional startup failure for SRE-agent validation #298)" (Revert SRE drill: restore AlitaBot startup #302)
• 6ab3c61 DRILL (revert after): AlitaBot intentional startup failure for SRE-agent validation (DRILL (revert after): AlitaBot intentional startup failure for SRE-agent validation #298)

These changes mention an intentional "DRILL" runtime failure and there was an automated ArgoCD deploy at 2026-07-27T11:01:56Z (revision 60005602) immediately before the first observed error sample at 2026-07-27T11:02:14Z.

Classification: correlated — suspected bad deploy (error-rate finding). Filed as priority-high.

[Szer]

Resolved — the cause was reverted.

Program.DrillRuntimeFailureLoop was an authorized SRE validation drill (#303), reverted in #304. Zero SRE_DRILL lines have appeared in Loki since the revert deployed; the elevated log_errors_24h (83) is the trailing 24h window still containing the drill period, and will decay out.

This issue is a success case, not noise. It is the first anomaly the monitor ever filed correctly, and it validated four of the six fixes from #307 on a real signal:

• dormant_exempt: false on log_errors_24h — the dormant carve-out no longer suppresses error series (defect 2)
• "correlated — suspected bad deploy… do NOT suppress" — attribution no longer converts a deploy-correlated error burst into "expected" (defect 3)
• Named Program.DrillRuntimeFailureLoop as a group unseen in 28 days (defect 4)
• Readable message prefix rather than a raw JSON blob (defect 5)

For contrast, the same underlying event during drill 2 produced: "attributed — the increased error count is plausibly explained by the recent intentional DRILL changes... that makes the observed error volume expected" — and nothing was filed while the bot was dead.

It also correctly overrode the mechanical p1=false gate, reasoning that the error-rate rule mandates priority-high regardless. That is the right relationship between a coarse mechanical flag and a judgement rule.