Deploy failure: alita-bot (0701b55)

[github-actions]

Deploy Failure

Bot: alita-bot
ArgoCD app: alita-bot
Container label: alita-bot
GHCR image: ghcr.io/szer/alita-bot
Commit: 0701b55
Workflow run: https://github.com/Szer/bots/actions/runs/30256392998
Failure class: app

Follow the runbook in .github/prompts/sre.md. Identify which verify-deploy.sh phase failed from the run logs above.

[github-actions]

Incident Report

Summary

• Severity: P2 (new ReplicaSet failing, old replicas still serving — no user impact)
• Bot: alita-bot
• Duration: No user-facing outage (new pods CrashLoopBackOff; existing replica(s) healthy)
• Root cause: Deliberate SRE validation drill present in commit 0701b55af8dd59f9d32bc5dca7166958634b6b8c that throws SRE_DRILL_INTENTIONAL_STARTUP_FAILURE on startup, causing the new pod to CrashLoopBackOff.

Timeline

1. Deploy triggered by commit 0701b55af8dd59f9d32bc5dca7166958634b6b8c (workflow: https://github.com/Szer/bots/actions/runs/30256392998)
2. verify-deploy.sh reported failure because the new pod did not reach ready (pod readiness/backoff). (Failure class: app)
3. ArgoCD: application alita-bot is Synced and Progressing (new ReplicaSet b7795b447 is Progressing, waiting for rollout).
4. Resource tree shows at least one healthy existing pod alita-bot-55d8cd7d87-wxcqg serving traffic; new pod alita-bot-b7795b447-bzbd7 is Degraded / CrashLoopBackOff.
5. Loki logs for the failing pod contain the following repeated startup failure:

Unhandled exception. System.Exception: SRE_DRILL_INTENTIONAL_STARTUP_FAILURE: this is a deliberate, owner-authorized SRE-agent validation drill forcing AlitaBot to crashloop on startup — it must be reverted immediately after the drill completes.

Diagnostics

• ArgoCD status: sync: Synced, health: Progressing (new ReplicaSet not available yet)
• Pods / ReplicaSets: existing ReplicaSet alita-bot-55d8cd7d87 Healthy and serving; new ReplicaSet alita-bot-b7795b447 Progressing with 0/1 new replicas available.
• Prometheus: failing pod alita-bot-b7795b447-bzbd7 shows CrashLoopBackOff; restart count observed (up to 5 restarts in recent window).
• Loki: failing pod logs show the SRE_DRILL_INTENTIONAL_STARTUP_FAILURE exception (stacktrace present). No sustained JSON level="Error" bursts were detected in the last 5 minutes.

Resolution

• No rollback performed — not required. Existing replicas remained healthy and continued serving traffic (P2).
• Action required: revert or remove the intentional SRE drill code in commit 0701b55af... (do not re-deploy the drill to production). After reverting, re-run the deploy or trigger ArgoCD sync.
• Auto-sync status: unchanged (automated sync remains enabled). No change made to auto-sync during this investigation.

Follow-up / Recommendations

• Revert the drill commit or restrict drill delivery to non-production environments only.
• Add a gating policy to prevent owner-authorized drills from being merged/deployed to production (e.g. feature flag, review-label, or CI check).
• Consider adding a short note in deploy-failure incidents when the failure contains an explicit SRE_DRILL_ marker.

Evidence attached: Loki excerpt (failing pod logs) and ArgoCD resource-tree showing healthy old pod + failing new pod.