Phase 6 — Event-Initiated Remediation
Closes the last gap in the AI-SRE loop: a detected anomaly auto-triggers the existing closed-loop change pipeline, risk-gated — no human button-press for LOW-risk events, one-click approval for MEDIUM/HIGH, page-out only for CRITICAL.
Risk model
- LOW (soft-clear a flapping BGP peer, clear counters) → auto-executes
- MEDIUM / HIGH (MTU restore, max-prefix guard, storm-control) → queues for one-click human approval
- CRITICAL → webhook page-out; never auto-acts
- Blast-Radius BFS escalation — ≥3 downstream devices bumps the tier +1, ≥8 bumps +2 (capped at CRIT, never demotes the floor)
- Config changes flow through the 6-stage closed loop with confirmed-commit (RFC 6241 §8.4) — the device auto-reverts if health degrades in the watch window
What's in it
src/auto_remediate.py— dependency-injected engine (load · normalize · match · fill · tier · decide · execute · queue · loop)src/runbooks/auto.yaml— 8-runbook catalog (3 LOW / 3 MED / 2 HIGH)- 6 API endpoints under
/api/auto-remediate/*(status, queue, runbook, approve, decline, simulate) - Auto-Remediation Queue UI tab in the demo dashboard
- 5 new
mv_auto_*MCP tools → ~68 tools total for Claude Code / Cursor
Security hardening
simulateroute off by default (only registered withMVLAB_AUTO_REMEDIATE_SIMULATE=1), behind theX-API-Keygate- Placeholder injection guard — every
{placeholder}strictly validated byvalid_field()before substitution; bad values parkrejected_invalid_fieldand never execute - Decline audit-integrity guard — only cancels a run still awaiting a verdict
- Mutating endpoints gated by
MVLAB_API_KEY; loopback-bound, fail-closed - Tests: 34/34 (27 auto-remediate + 7 MCP smoke)
Also since v0.5.0
- ChatOps Telegram bot — ask the fabric from chat
- Orchestrator — vendor-aware live collection for containerlab nodes, device-group resolution + fleet fallback for
/ask - Security —
DCN_API_KEY→MVLAB_API_KEYrename across client/bot surfaces; ultrareview hardening - Docs — animated one-page documentation site + architecture guide
Lab unchanged: 26 nodes · 4 vendors (Juniper, Arista, Nokia SR Linux, FRR) · real BGP + EVPN/VXLAN on containerlab · single laptop · MIT · no cloud.