AI-powered incident detection and root cause analysis for Kubernetes service meshes
Reduced incident response time from ~60 minutes of manual debugging to ~25 minutes of automated investigation β just read the Slack alert and run the suggested kubectl command.
- Overview
- The Problem
- How It Works
- Architecture
- RCA Agent Pipeline
- Tech Stack
- Related Repositories
- Project Structure
- Prerequisites
- Quick Start
- Environment Variables
- NATS Subjects
- Neo4j Graph Schema
- Istio Flags Detected
- Traffic Alert Schema
- Slack Alert Output
- CI/CD
- Contributing
Service Mesh RCA is an event-driven, AI-powered system that autonomously detects, investigates, and diagnoses failures in Kubernetes service mesh environments. It continuously observes your Istio service mesh through Kiali, builds a live topology graph in Neo4j, detects traffic anomalies (like UH, UF, NR flags), and triggers a sequential AI agent pipeline built on Google ADK that performs root cause analysis β delivering a full incident report with remediation commands directly to Slack.
No dashboards to watch. No logs to grep. Just read the Slack alert and run the command.
When a service fails in a Kubernetes mesh:
| Without RCA System | With RCA System |
|---|---|
| β±οΈ ~60 min manual investigation | β‘ ~25 min automated analysis |
| π Manually check Kiali dashboard | π€ Auto-detects Istio failure flags |
| π Grep through pod logs | π AI agent fetches & analyzes logs |
| π§© Trace upstream/downstream manually | π Neo4j graph topology auto-queried |
| π€ Guess root cause from experience | π§ Sequential AI agents deduce root cause |
| π¬ Manually write incident report | π¬ Rich Slack incident card with kubectl fix |
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CONTINUOUS LOOP (every 20s) β
β β
β Istio Mesh βββΊ Kiali API βββΊ Observer Agent (this repo) β
β β β
β βββββββββββββ΄βββββββββββββ β
β βΌ βΌ β
β graph.snapshot traffic.alert β
β (NATS JS) (NATS JS) β
β β β β
β βΌ βΌ β
β Memory Engine RCA Agent (ADK) β
β β β β
β βΌ β β
β Neo4j Graph DB ββββββββββββββββββ β
β β β
β βΌ β
β Slack Alert π β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Step-by-step flow:
- Observer Agent (this repo) polls the Kiali API every 20 seconds to fetch the latest service mesh graph
- The graph (with pod metadata from the Kubernetes API) is published to NATS JetStream on subject
graph.snapshot - The Observer also analyzes edges for Istio failure flags (
UH,UF,NR) β when detected, aTrafficAlertis published totraffic.alert - The Memory Engine subscribes to
graph.snapshotand stores the topology in Neo4j with relationships:Service -[TRAFFIC_TO]β ServiceandService -[HAS_POD]β Pod - The RCA Agent (Google ADK) subscribes to
traffic.alertand kicks off a sequential AI pipeline:- Analytics Agent β queries Neo4j for topology, blast radius, symptoms
- Flag Router β routes to specialized agent based on the Istio flag
- Infra Workload Agent β checks pod status via NATS request/reply
- On-Demand Logs Agent β fetches pod logs via NATS request/reply
- Final RCA Agent β synthesizes all findings into a root cause report
- Notification β posts incident card to Slack
The system is composed of 4 microservices and 2 databases, connected through NATS JetStream as the central event bus:
| Component | Role | Technology |
|---|---|---|
| Observer Agent (this repo) | Polls Kiali, enriches graph with pod data, detects anomalies, publishes events | Go, Gin, K8s client-go |
| NATS JetStream | Durable message bus for graph snapshots and traffic alerts | NATS with JetStream persistence |
| Memory Engine | Subscribes to graph snapshots, builds/updates Neo4j topology | Node.js / Go |
| Neo4j | Graph database storing service topology and pod relationships | Neo4j 5.x |
| RCA Agent | Sequential AI agent pipeline for root cause analysis | Python, Google ADK (Gemini) |
| Slack | Incident notification delivery | Slack Webhooks |
The RCA Agent is a sequential workflow built with Google Agent Development Kit (ADK) that orchestrates 5 specialized sub-agents:
graph LR
A[π¨ Traffic Alert] --> B[π Analytics Agent]
B --> C{π Flag Router}
C -->|UH/UF/UC| D[ποΈ Infra Workload Agent]
C -->|NR/NC| E[π Mesh Config Agent]
D --> F[π On-Demand Logs Agent]
E --> F
F --> G[π¬ Final RCA Agent]
G --> H[π Slack Notification]
| Agent | Trigger | What It Does | Tools |
|---|---|---|---|
| Analytics Agent | Every traffic alert | Queries Neo4j to find the failed pod, upstream/downstream services, blast radius, and deduces symptoms from the alert metadata | read_neo4j_cypher, get_current_time |
| Infra Workload Agent | Flags: UH, UF, UC |
Checks pod status via NATS request/reply to the Observer, detects CrashLoopBackOff, OOMKilled, scheduling failures | check_pod_status_via_nats |
| Mesh Config Agent | Flags: NR, NC |
Analyzes Istio VirtualServices, DestinationRules, and K8s Service configurations for routing misconfigurations | β |
| On-Demand Logs Agent | After triage agent | Fetches container logs via NATS request/reply, analyzes for stack traces and application errors | fetch_pod_logs_via_nats |
| Final RCA Agent | After all data gathered | Synthesizes all findings from prior agents into a cohesive root cause analysis with remediation plan | β |
| Notification Action | After RCA complete | Deterministically posts a rich incident card to Slack with severity, blast radius, and kubectl command |
send_slack_incident_alert |
The Observer Agent (this repo) also acts as a Kubernetes API bridge via NATS request/reply:
RCA Agent βββΊ NATS "observer.pod.status" βββΊ Observer Agent βββΊ K8s API βββΊ Response
RCA Agent βββΊ NATS "observer.pod.logs" βββΊ Observer Agent βββΊ K8s API βββΊ Response
This allows the RCA Agent (running in Python) to query Kubernetes pod status and logs without needing direct cluster access.
| Layer | Technology | Purpose |
|---|---|---|
| Language | Go 1.25 | Observer Agent core |
| Web Framework | Gin | HTTP server & health endpoints |
| Messaging | NATS + JetStream | Event-driven communication |
| Graph Database | Neo4j | Service topology storage |
| Container Runtime | Docker + Distroless | Minimal, secure container images |
| Kubernetes | client-go | Pod status, logs, events API |
| Service Mesh | Istio | Traffic management & mTLS |
| Observability | Kiali API | Service mesh visualization & graph |
| AI Framework | Google ADK (Python) | Sequential agent orchestration |
| AI Model | Google Gemini | LLM for root cause reasoning |
| Notifications | Slack Webhooks | Incident alerting |
| CI/CD | GitHub Actions | Build, scan (Trivy), push to GAR |
This project is composed of 3 repositories that work together:
| Repository | Description | Tech |
|---|---|---|
| Service-Mesh (this repo) | Observer Agent β polls Kiali, publishes graph snapshots & traffic alerts to NATS, serves as K8s API bridge | Go |
| golang-memory-engine | Subscribes to graph.snapshot via NATS, builds and maintains the service topology graph in Neo4j |
Go |
| RCA-pipeline | Sequential AI agent pipeline (Google ADK) β consumes traffic.alert, queries Neo4j, investigates pods, produces RCA report, sends Slack notification |
Python |
Service-Mesh/
βββ cmd/
β βββ server/
β βββ main.go # Entry point β HTTP server, NATS connection, JetStream setup
β βββ main_test.go # Server tests
βββ istio/
β βββ istio.go # Core logic β Kiali polling, graph publishing, alert detection,
β β # pod status observer, pod logs observer
β βββ istio_test.go # Istio handler tests
βββ model/
β βββ graph.go # Data models β GraphResponse, TrafficAlert, PodAlert, etc.
βββ docs/
β βββ images/ # Architecture & pipeline diagrams
βββ .github/
β βββ workflows/
β βββ ci.yml # CI pipeline β build, Trivy scan, push to GCP Artifact Registry
βββ Dockerfile # Multi-stage build (Go 1.25 β Distroless)
βββ rca-agent.yaml # Docker Compose β all services (observer, NATS, Neo4j, memory engine, RCA agent)
βββ .env.example # Environment template
βββ go.mod # Go module dependencies
βββ go.sum # Dependency checksums
Before running the project, make sure you have:
| Requirement | Version | Purpose |
|---|---|---|
| Docker & Docker Compose | v20+ / v2+ | Run all services |
| Kubernetes Cluster | v1.28+ | Running your microservices with Istio |
| Istio | v1.20+ | Service mesh with mTLS and traffic management |
| Kiali | v1.70+ | Istio observability dashboard (provides the graph API) |
| kubectl | v1.28+ | Cluster access (mounted into observer container) |
| Go | v1.25+ | (only if building locally) |
| Slack Webhook URL | β | For receiving incident notifications |
| Google API Key | β | For Gemini LLM access in RCA Agent |
Caution
Istio is mandatory. Without Istio installed in your cluster, the Kiali API will have no data and the entire RCA pipeline will not function. NATS is also required as the central message bus.
# Install KinD if not already installed
# https://kind.sigs.k8s.io/docs/user/quick-start/#installation
go install sigs.k8s.io/kind@latest
# Create a cluster (use a config for extra ports if needed)
cat <<EOF | kind create cluster --name rca-mesh --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30000
hostPort: 30000
protocol: TCP
- role: worker
- role: worker
EOF
# Verify the cluster is running
kubectl cluster-info --context kind-rca-mesh# Download and install istioctl
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.24.0 sh -
cd istio-1.24.0
export PATH=$PWD/bin:$PATH
# (Optional) Add istioctl to your shell permanently
echo 'export PATH="$HOME/istio-1.24.0/bin:$PATH"' >> ~/.bashrc
source ~/.bashrcChoose an Istio installation profile based on your needs:
| Profile | Components | Use Case |
|---|---|---|
demo |
istiod, ingress gateway, egress gateway | β Recommended for this project β includes everything |
default |
istiod, ingress gateway | Production with minimal footprint |
minimal |
istiod only | Testing Istio control plane |
# Install Istio with the demo profile (recommended)
istioctl install --set profile=demo -y
# Verify Istio installation
istioctl verify-install
# Enable automatic sidecar injection for the default namespace
kubectl label namespace default istio-injection=enabled
# Verify all Istio components are running
kubectl get pods -n istio-systemExpected output β all pods should be Running:
NAME READY STATUS RESTARTS AGE
istio-egressgateway-xxx 1/1 Running 0 2m
istio-ingressgateway-xxx 1/1 Running 0 2m
istiod-xxx 1/1 Running 0 2m
Important
Wait until all Istio pods (istiod, istio-ingressgateway, istio-egressgateway) show Running before proceeding. Use kubectl get pods -n istio-system -w to watch.
Kiali requires Prometheus to function. We also recommend installing Grafana and Jaeger for full observability:
# Install all Istio addons (Prometheus, Kiali, Grafana, Jaeger)
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/jaeger.yaml
# Wait for all addons to be ready
kubectl rollout status deployment/kiali -n istio-system
kubectl rollout status deployment/prometheus -n istio-system
# Port-forward Kiali to localhost:20001
kubectl port-forward svc/kiali -n istio-system 20001:20001 &Verify Kiali is accessible at: http://localhost:20001
Note
The Observer Agent in this project calls the Kiali API at http://localhost:20001/kiali/api/namespaces/graph to fetch the service mesh graph. Make sure port-forwarding is active or configure BASE_URL in .env accordingly.
This project uses Google Online Boutique β a cloud-native, polyglot microservices application with 11 services that generates rich inter-service traffic, perfect for observing the full mesh topology in Kiali.
βββββββββββββββββ
β Frontend β
ββββββββ¬βββββββββ
βββββββββββββββββΌββββββββββββββββ
βΌ βΌ βΌ
ββββββββββββββ ββββββββββββββ ββββββββββββββββ
β Ad Service β βCart Serviceβ βProduct Catalogβ
ββββββββββββββ βββββββ¬βββββββ ββββββββββββββββ
βΌ
ββββββββββββββ
β Checkout β
ββββββββ¬ββββββ
βββββββββββββββββ¬ββββΌββββ¬βββββββββββββββββ
βΌ βΌ βΌ βΌ βΌ
βββββββββββββ ββββββββββββ βββββββββ ββββββββββββββββ
β Shipping β β Payment β β Email β β Currency β
βββββββββββββ ββββββββββββ βββββββββ ββββββββββββββββ
ββββββββββββββββββββ
β Recommendation β
ββββββββββββββββββββ
ββββββββββββββββββββ
β Load Generator β
ββββββββββββββββββββ
# Clone the Google microservices demo
git clone https://github.com/GoogleCloudPlatform/microservices-demo.git
cd microservices-demo
# Deploy all 11 microservices to the default namespace (with Istio sidecar injection)
kubectl apply -f ./release/kubernetes-manifests.yaml
# Wait for all pods to be ready (each pod should show 2/2 β app container + Istio sidecar)
kubectl get pods -wExpected output β all pods should show 2/2 Running:
NAME READY STATUS RESTARTS AGE
adservice-xxx 2/2 Running 0 3m
cartservice-xxx 2/2 Running 0 3m
checkoutservice-xxx 2/2 Running 0 3m
currencyservice-xxx 2/2 Running 0 3m
emailservice-xxx 2/2 Running 0 3m
frontend-xxx 2/2 Running 0 3m
loadgenerator-xxx 2/2 Running 0 3m
paymentservice-xxx 2/2 Running 0 3m
productcatalogservice-xxx 2/2 Running 0 3m
recommendationservice-xxx 2/2 Running 0 3m
shippingservice-xxx 2/2 Running 0 3m
Tip
The loadgenerator service automatically sends continuous traffic across all services, so you'll see a rich traffic graph in Kiali and meaningful data in the Observer Agent's graph snapshots within ~60 seconds of deployment.
The NATS CLI is useful for debugging and monitoring messages flowing through the system:
# Install NATS CLI
# Option 1: Go install
go install github.com/nats-io/natscli/nats@latest
# Option 2: Download binary (Linux amd64)
curl -L https://github.com/nats-io/natscli/releases/latest/download/nats-0.1.5-linux-amd64.zip -o nats.zip
unzip nats.zip -d /usr/local/bin/
# Option 3: Homebrew (macOS/Linux)
brew tap nats-io/nats-tools
brew install nats-io/nats-tools/natsAfter all services are running, you can use NATS CLI to monitor traffic:
# Subscribe to graph snapshots (see what the Observer is publishing)
nats sub graph.snapshot --server=nats://localhost:4222
# Subscribe to traffic alerts (see detected failures)
nats sub traffic.alert --server=nats://localhost:4222
# Check JetStream stream info
nats stream info GRAPH --server=nats://localhost:4222
nats stream info TRAFFIC --server=nats://localhost:4222Note
The NATS server itself is included in the Docker Compose file (rca-agent.yaml) β you don't need to install it separately. The NATS CLI above is only for debugging/monitoring from your host machine.
# Clone the Observer Agent (this repo)
git clone https://github.com/TechNinjaAyush/Service-Mesh.git
cd Service-Mesh
# Clone the Memory Engine
git clone https://github.com/TechNinjaAyush/golang-memory-engine.git
# Clone the RCA Agent
git clone https://github.com/TechNinjaAyush/RCA-pipeline.git# Copy the example env file
cp .env.example .envEdit .env with your configuration:
graphType=app
namespaces=default
duration=60s
BASE_URL=http://host.docker.internal:20001/kiali/api/namespaces/graph
NATS_URL=nats://nats:4222# Build the Observer Agent
docker build -t service-mesh-new:v1.2.3 .
# Build the Memory Engine (from its repo)
cd ../golang-memory-engine
docker build -t memory-engine:v1.1.4 .
# Build the RCA Agent (from its repo)
cd ../RCA-pipeline
docker build -t rca-agent:v1.2.3 .# If using KinD (Kind in Docker)
docker network create kind 2>/dev/null || trueEdit rca-agent.yaml and replace the placeholder values:
# In the rca-agent service:
environment:
GOOGLE_API_KEY: "your-actual-google-api-key"
SLACK_WEBHOOK_URL: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
# In the neo4j service:
environment:
NEO4J_AUTH: neo4j/your-secure-password
# In the memory-engine service:
environment:
NEO4J_PASSWORD: "your-secure-password"docker compose -f rca-agent.yaml up -d# Check all containers
docker compose -f rca-agent.yaml ps
# Check Observer Agent health
curl http://localhost:8080/health
# Check NATS server
curl http://localhost:8222/varz
# Check Neo4j browser
# Open http://localhost:7474 in your browser# Observer Agent logs (see Kiali polling & alert detection)
docker logs -f service-mesh-new
# RCA Agent logs (see AI pipeline execution)
docker logs -f rca-agent
# Memory Engine logs (see graph snapshots being stored)
docker logs -f memory-engine-new| Variable | Description | Default |
|---|---|---|
BASE_URL |
Kiali API graph endpoint | http://localhost:20001/kiali/api/namespaces/graph |
namespaces |
Kubernetes namespaces to monitor | default |
graphType |
Kiali graph type (app, workload, service) |
app |
duration |
Time window for Kiali metrics | 60s |
NATS_URL |
NATS server connection URL | nats://localhost:4222 |
| Variable | Description |
|---|---|
GOOGLE_API_KEY |
Google Gemini API key for LLM reasoning |
NATS_URL |
NATS server URL |
NEO4J_URI |
Neo4j Bolt protocol URI |
NEO4J_USER |
Neo4j username |
NEO4J_PASSWORD |
Neo4j password |
NATS_SUBJECT |
NATS subject to subscribe (traffic.alert) |
SLACK_WEBHOOK_URL |
Slack incoming webhook URL |
| Variable | Description |
|---|---|
NEO4J_URI |
Neo4j Bolt protocol URI |
NEO4J_USERNAME |
Neo4j username |
NEO4J_PASSWORD |
Neo4j password |
NATS_URL |
NATS server URL |
| Subject | Publisher | Consumer | Payload | Persistence |
|---|---|---|---|---|
graph.snapshot |
Observer Agent | Memory Engine | Full Kiali graph JSON with pod metadata | JetStream (WorkQueue) |
traffic.alert |
Observer Agent | RCA Agent | TrafficAlert JSON |
JetStream (WorkQueue) |
observer.pod.status |
RCA Agent (request) | Observer Agent (reply) | Pod status with container details, events, service match | Core NATS (Request/Reply) |
observer.pod.logs |
RCA Agent (request) | Observer Agent (reply) | Container logs (last 50 lines per container) | Core NATS (Request/Reply) |
The Memory Engine stores the service mesh topology with two relationship types:
-- Service-to-Service traffic relationship
(s1:Service {app: "frontend"})-[:TRAFFIC_TO]->(s2:Service {app: "backend"})
-- Service-to-Pod ownership relationship
(s:Service {app: "frontend"})-[:HAS_POD]->(p:Pod {name: "frontend-849f6b48f8-v6j2q"})-- Find all pods for a failed service
MATCH (s:Service)-[:HAS_POD]->(p:Pod)
WHERE s.app = "frontend"
RETURN p.name AS failed_pod LIMIT 1
-- Find upstream/downstream services and blast radius
MATCH (failed:Service)
WHERE failed.app = "frontend"
OPTIONAL MATCH (up:Service)-[:TRAFFIC_TO]->(failed)
WITH failed, collect(DISTINCT up.app) AS upstream_services
OPTIONAL MATCH (failed)-[:TRAFFIC_TO]->(down:Service)
RETURN
upstream_services,
collect(DISTINCT down.app) AS downstream_services,
CASE WHEN size(upstream_services) > 0 THEN size(upstream_services) ELSE 1 END AS blast_radiusThe Observer Agent monitors Kiali edge traffic for these Istio response flags:
| Flag | Meaning | Routed To |
|---|---|---|
UH |
No healthy upstream β all upstream hosts are unhealthy | Infra Workload Agent |
UF |
Upstream connection failure β connection to upstream failed | Infra Workload Agent |
UC |
Upstream connection termination β connection was terminated | Infra Workload Agent |
NR |
No route configured β no matching route found | Mesh Config Agent |
NC |
No cluster found β cluster lookup failed | Mesh Config Agent |
When a failure flag is detected, the Observer publishes this structure to traffic.alert:
{
"incidentId": "550e8400-e29b-41d4-a716-446655440000",
"source": "frontend",
"target": "backend",
"responseCode": "503",
"flag": "UH",
"host": "backend",
"protocol": "http",
"responseTime": "245ms",
"rates": {
"http": "0.50",
"httpPercentErr": "100.00"
},
"flagPercent": "100.00",
"isMTLS": "true",
"message": "Traffic failure detected from frontend to backend (Host: backend) with protocol http, flag UH (100.00%), code 503",
"timestamp": 1722950824
}The final RCA report is posted to Slack as a rich incident card containing:
- π Incident ID β unique UUID
- π΄ Severity β critical / high / medium (based on blast radius)
- π₯ Failed Service & Pod β exact pod name
- π Root Cause β AI-deduced root cause
- π Diagnosis Summary β detailed analysis
- π Impacted Services β upstream/downstream dependencies
- π£ Blast Radius β number of affected services
- π οΈ Remediation β ready-to-run
kubectlcommand
Example:
kubectl rollout restart deployment frontend -n default
or for scaled-to-zero scenarios:
kubectl scale deployment frontend --replicas=1 -n default
The project uses GitHub Actions for continuous integration:
# .github/workflows/ci.yml
Trigger: Push to main / Manual dispatch
Pipeline:
1. Checkout repository
2. Authenticate to Google Cloud (Workload Identity Federation)
3. Setup gcloud SDK
4. Configure Docker for GCP Artifact Registry
5. Build Docker image (multi-stage, distroless)
6. Scan with Trivy (CRITICAL + HIGH vulnerabilities)
7. Push to Google Artifact Registry| Secret | Description |
|---|---|
GCP_PROJECT_ID |
Google Cloud project ID |
GCP_REGION |
GCP region for Artifact Registry |
GAR_REPOSITORY |
Artifact Registry repository name |
WORKLOAD_IDENTITY_PROVIDER |
GCP Workload Identity provider |
SERVICE_ACCOUNT |
GCP service account email |
Contributions are welcome! Here's how to get started:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
# Run locally (requires NATS and Kiali running)
go run ./cmd/server
# Run tests
go test ./...
# Build binary
go build -o graph-publisher ./cmd/serverBuilt with β€οΈ for SRE teams who are tired of 3 AM debugging sessions
β Star this repo β’ π Report Bug β’ π‘ Request Feature

