Production-grade multi-tenant observability platform running entirely on a Mac M2 Pro. Zero cloud spend.
┌─────────────────────────────────────────────────────────────────────────────┐
│ SentinalFlow — Local Stack │
│ │
│ ┌──────────────┐ ┌─────────────────────────────────────────────────┐ │
│ │ Producers │ │ Kafka Layer (Redpanda) │ │
│ │ │ │ │ │
│ │ producer.py ├───►│ tenant-a-metrics (6 partitions) │ │
│ │ (tenant-a) │ │ tenant-b-metrics (6 partitions) │ │
│ │ (tenant-b) ├───►│ tenant-a-logs (3 partitions) │ │
│ │ │ │ tenant-b-logs (3 partitions) │ │
│ │ --breach │ │ slo-aggregates (6 partitions) ◄─────────┐ │ │
│ └──────────────┘ └──────────────────────┬──────────────────────┘ │ │
│ │ │ │
│ ▼ │ │
│ ┌──────────────────────────────────────────────────────────────────┐ │ │
│ │ Spark Structured Streaming │ │ │
│ │ │ │ │
│ │ spark-master + 2 workers │ │ │
│ │ │ │ │
│ │ tenant-{a,b}-metrics │ │ │
│ │ │ │ │ │
│ │ ▼ │ │ │
│ │ 1-min tumbling window, per tenant+service │ │ │
│ │ │ │ │ │
│ │ ├── avg_rps, avg_error_rate, avg_p99_ms │ │ │
│ │ └── slo_burn_rate = error_rate / 0.001 ──────────────────┐ │ │ │
│ │ │ │ │ │
│ │ Sink 1: s3a://sf-slo-aggregates/ (Parquet) ◄──────LocalStack │ │ │ │
│ │ Sink 2: slo-aggregates topic ──────────────────────────────────┼──┘ │ │
│ └─────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ Observability Stack (k3d) │ │
│ │ │ │
│ │ metrics_bridge.py │ │
│ │ slo-aggregates ──► Pushgateway ──► Prometheus ──► Grafana │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ AlertManager │ │
│ │ │ │ │
│ │ SLOErrorRateBreach (>1%) │ │
│ │ SLOFastBurn (burn>14.4) ──► rollback_ │ │
│ │ SLOP99LatencyBreach webhook.py │ │
│ │ │ │ │
│ └─────────────────────────────────────────────────────────────┼────────┘ │
│ │ │
│ ┌──────────────────────────────────────────────────────────────▼────────┐ │
│ │ GitOps Layer (ArgoCD) │ │
│ │ │ │
│ │ ApplicationSet ──► tenant-a (namespace, RBAC, NetworkPolicy, HPA) │ │
│ │ ──► tenant-b (namespace, RBAC, NetworkPolicy, HPA) │ │
│ │ │ │
│ │ selfHeal: true ── every git push auto-syncs │ │
│ │ rollback webhook ── SLO breach → argocd app rollback │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ AWS Emulation (LocalStack :4566) │ │
│ │ │ │
│ │ S3: sf-metrics-tenant-{a,b} sf-slo-aggregates sf-spark-chkpts │ │
│ │ IAM: spark-execution role tenant-{a,b}-readonly policies │ │
│ │ CloudWatch: /sentinalflow/{tenant}/app log groups + SLO alarms │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────────┘
| Service | Host Port | Notes |
|---|---|---|
| LocalStack | 4566 | AWS API endpoint |
| Redpanda | 19092 | Kafka bootstrap |
| Redpanda Console | 8080 | Kafka UI |
| Redpanda Admin | 9644 | Health / admin |
| Spark Master UI | 8888 | Job dashboard |
| Spark Job UI | 4040 | Active job detail |
| Grafana | 30300 (NodePort) | Tenant dashboards |
| Prometheus | 30900 (NodePort) | Metrics store |
| AlertManager | 30903 (NodePort) | SLO breach alerts |
| ArgoCD | port-forward 8080→80 | GitOps UI |
| Pushgateway | 30091 (NodePort) | Spark→Prometheus bridge |
brew install docker k3d helm terraform argocd awscli
pip install confluent-kafka prometheus-client pytest ruff mypydocker compose up -d --waitStarts: LocalStack, Redpanda, Redpanda Console, Spark (master + 2 workers), bootstrap init containers.
cd terraform/aws && terraform init && terraform apply -auto-approveCreates: S3 buckets, IAM roles, CloudWatch log groups + SLO alarms.
bash scripts/k3d-setup.shcd terraform/k8s && terraform init && terraform apply -auto-approveCreates: tenant-a, tenant-b namespaces with RBAC and NetworkPolicies.
bash scripts/helm-install.shcd kafka-producers
pip install -r requirements.txt
python producer.py # normal traffic
python producer.py --breach # simulate SLO breachbash scripts/spark-submit-slo.shFirst run downloads ~150MB of JARs. Windows appear in console after ~90s.
cd kafka-producers && python metrics_bridge.pykubectl apply -f argocd/applicationset.yaml
kubectl get applications -n argocdpython argocd/rollback_webhook.py # listens :5001# Kafka topics
docker exec sf-redpanda rpk topic list --brokers localhost:9092
# S3 buckets
aws --endpoint-url http://localhost:4566 s3 ls
# Spark output in S3
bash scripts/spark-check-s3.sh
# k8s tenant apps
kubectl get pods -n tenant-a
kubectl get pods -n tenant-b
# ArgoCD apps
kubectl get applications -n argocd# Terminal 1 — trigger breach traffic
python kafka-producers/producer.py --breach --tenant tenant-a
# Terminal 2 — watch AlertManager fire
curl -s http://localhost:30903/api/v2/alerts | jq '.[] | .labels'
# Terminal 3 — watch rollback webhook trigger
python argocd/rollback_webhook.py- Add entry to
argocd/applicationset.yamlgenerators list:- tenant: tenant-c replicaCount: "2" memoryLimit: "128Mi"
- Add
"tenant-c"tovar.tenantsinterraform/aws/variables.tfandterraform/k8s/variables.tf git commit && git push→ ArgoCD auto-deploys, Terraform provisions S3/IAM/CloudWatch on next apply
- Multi-tenant Kafka topic isolation at scale — per-tenant topics with separate partition groups, IAM policies enforce cross-tenant S3 isolation
- Spark Structured Streaming SLO aggregation pipeline — 1-min tumbling windows, watermark-based late-data handling, dual sink (S3 Parquet + Kafka)
- GitOps-driven tenant onboarding via ArgoCD ApplicationSet — add one YAML element to onboard a tenant; ApplicationSet auto-provisions namespace, RBAC, HPA, ServiceMonitor
- Automated rollback on SLO breach — AlertManager fires on error_rate >1% or burn_rate >14.4 → webhook calls
argocd app rollback
GitHub Actions on every push:
lint-python— ruff + mypylint-terraform— fmt check + validatelint-helm— helm lint + template both tenantstest-python— pytest unit testsdeploy— validates artifacts; ArgoCD self-heals viaselfHeal: true