Revert SRE drill: restore AlitaBot startup#302
Merged
Merged
Conversation
Szer
added a commit
that referenced
this pull request
Jul 27, 2026
…ent validation, take 2 (#303) Deliberate, owner-authorized production drill, second in the series (see #298/#302 for take 1 — a STARTUP failure that correctly did not trigger a rollback, since the old pod kept serving). This one is different on purpose: startup and /healthz succeed normally, so Kubernetes retires the old ReplicaSet and this broken one takes over — AlitaBot looks healthy to Kubernetes while being functionally dead, which should push the SRE agent toward a rollback this time. Two failure points, both logging SRE_DRILL_RUNTIME_FAILURE at Error via the existing Serilog logger, both gated on `not TestMode`: - DrillRuntimeFailureLoop: a hosted BackgroundService ticking every 15s, throwing and logging Error on every tick — traffic-independent, so it fires within verify-deploy.sh's 2-minute Loki error-log window even though AlitaBot is dormant and can go days without an inbound Telegram update. - The webhook update handler now throws before calling BotService.OnUpdate, so real Telegram updates fail too (secondary; the loop above is what actually gets detected). Gate verified live: prod bot_setting.TEST_MODE = 'false'; tests/AlitaBot.Tests sets TEST_MODE=true via both the container env var and a seeded bot_setting row (ContainerTestBase.fs). AlitaBot-only; no migration/config/Helm/workflow changes. REVERT IMMEDIATELY AFTER THE DRILL via `git revert` of this commit. Claude-Session: https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts the deliberate startup failure from #298. The drill is complete and succeeded.
Drill results
Two deploy runs were needed, and the first one found a real gap.
Run 1 (30255492040) —
verify-deploy.shcaught the crashloop and classified it correctly:Then
notify-failure: skipped,sre: skipped. Cause:alita-deploy.ymlhadsre-enabled: false, making AlitaBot the one bot in the fleet without SRE coverage — whilesre.mdclaimed "every bot ... opts into SRE coverage by default." Fixed in #300.Run 2 (30256392998) —
deploy / sre / investigate: success. The SRE agent executed for the first time in its existence and filed the incident report on #301.What the agent got right
if: always()re-enable was never neededDeclining the rollback is the important one: rolling back a P2 where the old pod is healthy would have been unnecessary disruption, and the agent has the ArgoCD permissions to do it.
Validated in production
verify-deploy.shinfra-vs-app classification (CI: actionlint gate + infra/app deploy-failure classification #279) against a real crashloop_bot-deploy.yml→_sre-agent.ymldirect invocation (Agent-flows Phase 0: fail-loud metrics, real SRE trigger, AlitaBot docs, orphan cleanup #274/Fix startup_failure: grant contents:write to _sre-agent.yml callers #275) — the fix for a trigger that had never once firedsre.md, including its "check for healthy old replicas before rolling back" instruction🤖 Generated with Claude Code
https://claude.ai/code/session_01P41roCWN5Fec7m69sSaunU