Skip to content

v0.6.0 — Phase 6: Event-Initiated Remediation

Latest

Choose a tag to compare

@gesh75 gesh75 released this 02 Jun 02:09

Phase 6 — Event-Initiated Remediation

Closes the last gap in the AI-SRE loop: a detected anomaly auto-triggers the existing closed-loop change pipeline, risk-gated — no human button-press for LOW-risk events, one-click approval for MEDIUM/HIGH, page-out only for CRITICAL.

Risk model

  • LOW (soft-clear a flapping BGP peer, clear counters) → auto-executes
  • MEDIUM / HIGH (MTU restore, max-prefix guard, storm-control) → queues for one-click human approval
  • CRITICAL → webhook page-out; never auto-acts
  • Blast-Radius BFS escalation — ≥3 downstream devices bumps the tier +1, ≥8 bumps +2 (capped at CRIT, never demotes the floor)
  • Config changes flow through the 6-stage closed loop with confirmed-commit (RFC 6241 §8.4) — the device auto-reverts if health degrades in the watch window

What's in it

  • src/auto_remediate.py — dependency-injected engine (load · normalize · match · fill · tier · decide · execute · queue · loop)
  • src/runbooks/auto.yaml8-runbook catalog (3 LOW / 3 MED / 2 HIGH)
  • 6 API endpoints under /api/auto-remediate/* (status, queue, runbook, approve, decline, simulate)
  • Auto-Remediation Queue UI tab in the demo dashboard
  • 5 new mv_auto_* MCP tools → ~68 tools total for Claude Code / Cursor

Security hardening

  • simulate route off by default (only registered with MVLAB_AUTO_REMEDIATE_SIMULATE=1), behind the X-API-Key gate
  • Placeholder injection guard — every {placeholder} strictly validated by valid_field() before substitution; bad values park rejected_invalid_field and never execute
  • Decline audit-integrity guard — only cancels a run still awaiting a verdict
  • Mutating endpoints gated by MVLAB_API_KEY; loopback-bound, fail-closed
  • Tests: 34/34 (27 auto-remediate + 7 MCP smoke)

Also since v0.5.0

  • ChatOps Telegram bot — ask the fabric from chat
  • Orchestrator — vendor-aware live collection for containerlab nodes, device-group resolution + fleet fallback for /ask
  • SecurityDCN_API_KEYMVLAB_API_KEY rename across client/bot surfaces; ultrareview hardening
  • Docs — animated one-page documentation site + architecture guide

Lab unchanged: 26 nodes · 4 vendors (Juniper, Arista, Nokia SR Linux, FRR) · real BGP + EVPN/VXLAN on containerlab · single laptop · MIT · no cloud.