Commandos: Agentic Incident Management

中文说明

Build a reproducible lab for agentic incident management on Kubernetes.

The lab should run locally on minikube for development and on Alibaba Cloud for cloud demonstrations. It should deploy sample services, observability tooling, incident injection tools, and a multi-agent incident response system in the same cluster.

Alibaba Cloud support is not implemented yet. The local minikube demo remains the default workflow, and an AWS EKS deployment overlay is available for running the same demo stack on an existing EKS cluster.

Demo

Use this workflow for the local demo. The Makefile accepts either ENV=local or the shorthand local, so make up local and make up ENV=local are equivalent.

If your network needs the local proxy, export it first:

export https_proxy=http://127.0.0.1:7897
export http_proxy=http://127.0.0.1:7897
export all_proxy=socks5://127.0.0.1:7897

Build or update the full local demo environment:

make up local MINIKUBE_HTTP_PROXY=http://host.minikube.internal:7897

This command runs the local setup as a single ordered pipeline:

1. Checks or installs minikube.
2. Checks or installs the selected minikube driver dependency.
3. Checks or installs Helm.
4. Starts the kuraudo-lab minikube profile.
5. Deploys the independent observability namespace.
6. Deploys the OpenTelemetry Demo in otel-demo and connects it to the observability stack.
7. Runs the demo and observability smoke checks.
8. Installs and builds the incident management React UI.

make up local does not create or reset incident records. To add local demo records explicitly, run make seed local.

After make up local succeeds, use separate terminals for foreground demo processes.

Forward the Kubernetes demo app and observability endpoints:

make ports-forward local

Start the local incident record API and web UI:

make serve local

Then use:

• Demo app: http://localhost:8080/
• Load Generator: http://localhost:8080/loadgen/
• Feature Flags: http://localhost:8080/feature/
• Grafana: http://localhost:3000/ with admin / admin
• Prometheus: http://localhost:9090/
• Mimir: http://localhost:9009/
• Incident record API: http://localhost:8090/
• Incident management UI: http://localhost:8091/

make ports-forward local and make serve local are intentionally separate because both are long-running foreground commands. Press Ctrl-C to stop each one.

To verify an already-running local demo without reinstalling or redeploying:

make verify local

Inject a named incident after the local demo is running:

chaos/incidents.sh inject frontend-service-selector-invalid
chaos/incidents.sh status frontend-service-selector-invalid

Recover it with:

chaos/recover-incidents.sh recover frontend-service-selector-invalid

List all supported scenarios with chaos/incidents.sh list. The full command matrix and expected Prometheus signals are documented in specs/stories/phase-1/004-add-incident-injection-tooling.md.

To tear down the local demo environment:

make down local

make down local stops the minikube profile. It does not remove local incident state.

AWS EKS Demo

Provision a default low-cost EKS cluster with Terraform:

make eks-infra-init
make eks-infra-plan
make eks-infra-apply

The Terraform code lives in infra/aws/eks. It defaults to ap-south-1 because AWS official Price List data checked on 2026-07-26 showed ap-south-1 and ap-south-2 tied as the cheapest eligible commercial EKS regions for this lab profile. ap-south-1 is used as the tie-breaker because it is the more established region. Override aws_region in Terraform if needed.

The Terraform-managed cluster includes:

• Public subnets and no NAT Gateway to keep PoC cost down.
• A managed ARM node group using t4g.small by default.
• EBS CSI and a default gp3 StorageClass.
• AWS Load Balancer Controller for NLB-backed Service objects.

After Terraform finishes, configure kubectl from the output command:

cd infra/aws/eks
terraform output kubeconfig_update_command

The EKS deployment workflow assumes your current kubectl context points to the cluster and the operator machine has Helm and kubectl.

Deploy to the current EKS context:

make up eks

This reuses the local Helm values and layers EKS overlays on top:

• apps/opentelemetry-demo/values-eks.yaml exposes frontend-proxy with an internet-facing AWS Network Load Balancer.
• observability/*/values-eks.yaml enables gp3 PVCs for Prometheus, Loki, Tempo, and Grafana.
• observability/mimir/mimir-eks.yaml replaces Mimir emptyDir storage with a gp3 PVC.

Get the public demo endpoint:

kubectl get svc frontend-proxy -n otel-demo

Grafana is intentionally kept as ClusterIP because the demo credentials are admin / admin. Use port-forwarding for access:

kubectl port-forward -n observability svc/observability-grafana 3000:80

Verify an existing EKS deployment:

make verify eks

Remove the deployed releases without deleting the cluster:

make down eks

Delete the demo namespaces after removing releases:

make destroy eks

Remove Terraform-managed infrastructure after the demo stack has been removed:

make eks-infra-destroy

Requirements

• The whole environment can be set up and torn down with one command to control cloud spend.
• Local development should work on minikube before using Alibaba Cloud.
• Each major component should live in its own subproject because this repository is a monorepo.
• Cloud infrastructure should stay cost-conscious; this repo currently includes AWS EKS Terraform and keeps Alibaba Cloud as planned future work.
• The system should be demonstrable end-to-end: deploy services, inject an incident, detect it, coordinate a response, and record status updates.

Proposed Subprojects

• infra: Terraform code for cloud resources and Kubernetes access.
• apps: sample applications and Helm values used as incident targets.
• observability: Prometheus and LGTM stack configuration.
• chaos: incident injection tools and scenarios.
• agents: multi-agent incident management system.
• platform: shared Makefile targets, scripts, and environment conventions.

Story Status

Phase 1

Story	Title	Status	Notes
0.1	Run the system locally on minikube	Done	Local minikube startup is now part of make up/status local with the qemu2 driver.
0.2	Provide one-command workflows with Makefile targets	Done	Local orchestration is consolidated into make up/down/status/verify/ports-forward/serve/agents/seed local; EKS orchestration is available for existing clusters, and Alibaba Cloud remains a phase 2 cloud story.
2	Deploy sample applications and services with Helm	Done	OpenTelemetry Demo deploys through Helm as part of make up local and is verified by make verify local.
3	Install observability with Prometheus and LGTM	Done	Independent observability namespace deploys Grafana, Loki, Tempo, Prometheus, Mimir, and an OTel collector through make up local; Prometheus remote-writes metrics to Mimir and Grafana queries Mimir.
4	Add incident injection tooling	Done	chaos/incidents.sh implements script-first incident injection and recovery; chaos/test-incidents.sh is verified locally.
5	Design the multi-agent incident management system	Done	Functional definition reviewed and split into delivery stories 8-13.
6	Demonstrate an end-to-end incident response	Done	Phase 1 local demo path covers setup, observability, incident injection, record API/UI, orchestrator loop, approvals, recovery, reports, and teardown.
8	Define agent system configuration	Done	YAML config, schema, local overlay, execution policy, and make validate are verified with uv and Python 3.12.
9	Define incident record format	Done	Format docs, complete example, validator, and make validate are reviewed.
10	Build incident record management service	Done	FastAPI service, Markdown storage adapter, append APIs, tests, and make serve local are reviewed.
11	Build multi-agent orchestrator	Done	Single-process orchestrator loop coordinates detector, Incident Commander, Operations Lead, Communications Lead, record service writes, advisory runbook context, and approved recovery execution.
12	Build incident management web UI	Done	Operator UI supports incident activity, approvals, manual incidents, post-mortems, runbook workflows, and loaded runbook context for the local demo.
13	Deploy and verify the agent system	Done	Phase 1 deploys and verifies the agent system as local foreground services with repository-controlled make serve/agents/seed local; cluster-hosted packaging moves to phase 2.
14	Generate Post-Mortem from Incident	Done	Record service and web UI can generate, view, and download deterministic Markdown post-mortem reports for resolved or closed incidents.
15	Generate KB Runbooks from Incident	Done	Record service and web UI can generate candidate runbooks, collect review comments, approve or request changes, refresh through pending diffs, render executable command hints, and download Markdown.
16	KB Runbooks Should Be Loaded	Done	Approved runbook context is retrieved by the record service, loaded into orchestrator investigation prompts as advisory context, recorded idempotently, and surfaced in the web UI.

Phase 2

Story	Title	Status	Notes
1	Provision a Kubernetes cluster on Alibaba Cloud	Planned	AWS EKS Terraform exists as an alternate cloud path; Alibaba Cloud remains planned.
7	Support Alibaba Cloud	Planned	Migration hardening story for all coding, optimization, and testing needed to move the local lab to Alibaba Cloud.