An agentic AI system that monitors real-time Kafka streams, detects anomalies autonomously, and either fixes them or escalates to a human - powered by 6 specialized AI agents.
Built by a senior backend engineer. No tutorials followed. Production-grade architecture.
Most pipeline failures don't announce themselves.
They show up as wrong numbers in a dashboard 3 hours later. Or a 2am page about data that's been corrupted since midnight.
StreamSentinel puts 6 AI agents on top of your Kafka streams to catch failures before any human notices, and either fix them automatically or wake up the right person with full context.
| Agent | Role | Tech |
|---|---|---|
| WatcherAgent | Monitors live Kafka streams for anomalies | Python, kafka-python |
| DiagnosisAgent | Uses LLM to explain root cause in plain English | Llama 3.2, Ollama |
| BlastRadiusAgent | Scores downstream impact deterministically | Python, BFS graph traversal |
| RemediationAgent | Auto-fixes LOW/MEDIUM, escalates HIGH to human | Python |
| NarratorAgent | Writes plain-English incident post-mortem | Llama 3.2, Markdown |
| MemoryAgent | Stores incidents in pgvector, retrieves similar past incidents as context | pgvector, PostgreSQL |
Live Kafka Stream
↓
WatcherAgent → detects anomaly
↓
MemoryAgent → retrieves similar past incidents as context
↓
DiagnosisAgent → local LLM explains root cause (no data leaves machine)
↓
BlastRadiusAgent → deterministic BFS scores impact LOW/MEDIUM/HIGH
↓
RemediationAgent → auto-fix or escalate to human
↓
NarratorAgent → writes incident post-mortem to disk
↓
MemoryAgent → stores incident for future context
↓
Back to watching...
Why deterministic blast radius scoring (no AI)?
The decision of whether to auto-fix or wake up a human must be predictable and auditable. LLMs introduce randomness. A BFS graph traversal doesn't. Governance requires determinism.
Why local Ollama (no cloud API)?
Financial transaction data never leaves the machine. Llama 3.2 runs locally via Ollama at localhost:11434. No Anthropic API. No OpenAI API. No cloud bill. No data risk.
Why Kafka-native (not Airflow)?
Every existing self-healing pipeline project wraps Airflow DAGs. StreamSentinel runs directly on Kafka consumer groups, watching live streams, not scheduled jobs. That's architecturally different and far more relevant to high-frequency financial data.
Why episodic memory (pgvector)?
Agents store past incidents as vector embeddings. When a new anomaly hits, similar past incidents are retrieved as context before diagnosis. The system gets smarter over time without retraining.
Why ThreadPoolExecutor for Ollama calls?
Ollama inference runs in a separate thread so it never blocks the Kafka consumer loop. A slow model response won't stall message processing or trigger a consumer group rebalance.
Streaming: Apache Kafka + Schema Registry
AI Agents: Custom Python agents + Llama 3.2 (Ollama - free, local)
Memory: pgvector + PostgreSQL
Observability: Prometheus + Grafana
Infrastructure: Docker + Docker Compose
Languages: Python
StreamSentinel exposes real-time metrics via Prometheus
and visualizes them in a live Grafana dashboard.
Health check endpoint available at http://localhost:8001/health.
Metrics tracked:
| Metric | What it measures |
|---|---|
streamsentinel_messages_total |
Messages processed per topic |
streamsentinel_anomalies_total |
Anomalies detected by type and severity |
streamsentinel_pipeline_duration_seconds |
Full pipeline processing time (MTTD) |
streamsentinel_active_anomalies |
Currently active anomalies |
streamsentinel_remediations_total |
Auto-remediations vs escalations |
View live dashboard: http://localhost:3000
- 🔴 LARGE_TRANSACTION - Suspicious transaction amount
- 🔴 SILENT_STREAM - No messages for 60+ seconds
- 🟡 RATE_DROP - Message rate drops 70%+ suddenly
- 🟡 SCHEMA_DRIFT - Upstream schema change breaks consumers
| Score | Meaning | Action |
|---|---|---|
| 🟢 LOW | Isolated impact | Auto-remediate silently |
| 🟡 MEDIUM | Analytics affected | Auto-remediate + notify team |
| 🔴 HIGH | Executive/ML/Compliance systems affected | Escalate to human immediately |
Service dependency graph is externalized in config/service_graph.json —
no code change needed to update service topology.
- Docker + Docker Compose
- Python 3.9+
- Ollama (for local LLM - free, no API key needed)
# Clone the repo
git clone https://github.com/Neelam95/StreamSentinel.git
cd StreamSentinel
# Start Kafka + Prometheus + Grafana
docker-compose up -d
# Install dependencies
pip install -r requirements.txt
# Pull the AI model (free, runs locally)
ollama pull llama3.2
# Start StreamSentinel
python main.py# Override database URL (default works for local Docker setup)
export DATABASE_URL=postgresql://streamsentinel:streamsentinel@localhost:5432/streamsentinelOpen http://localhost:3000 in your browser.
- Username:
admin - Password:
streamsentinel
Health check: http://localhost:8001/health
🔴🔴🔴 ANOMALY #1 - FULL PIPELINE STARTING
Step 1/4 - DiagnosisAgent diagnosing...
🧠 AI DIAGNOSIS COMPLETE
Root cause: Misconfigured payment gateway
Business impact: Regulatory exposure risk
Step 2/4 - BlastRadiusAgent scoring...
🔴 Blast Radius: HIGH
Affected: fraud-detection, accounting, compliance-reporting
Step 3/4 - RemediationAgent taking action...
🔴 HUMAN ESCALATION REQUIRED
⚠️ DO NOT AUTO-FIX - HUMAN DECISION REQUIRED
📟 ON-CALL ENGINEER PAGED
Step 4/4 - NarratorAgent writing report...
📰 INCIDENT POST-MORTEM REPORT saved to logs/
✅✅✅ ANOMALY #1 - PIPELINE COMPLETE
Duration: 35.65s
I am building StreamSentinel in public on LinkedIn. Follow the journey: Neelam Borse
Neelam Borse — Backend & Distributed Systems Engineer
- 5+ years building production data pipelines
- Currently @ Capital Group - Kafka, Spark, AWS at scale
- LinkedIn: linkedin.com/in/gauriborse
- GitHub: github.com/Neelam95