observ — self-hosted observability & SLO platform

A local Kubernetes observability stack that instruments a service, defines SLOs with multi-window burn-rate alerting, and proves detection works by injecting failures. Runs entirely on your machine via kind + Nix.

What it demonstrates

• RED (Rate, Errors, Duration) for the service; USE (Utilization, Saturation, Errors) for the nodes.
• SLIs / SLOs / error budgets with recording rules and an error-budget-remaining panel.
• Multi-window, multi-burn-rate alerting (Google SRE Workbook): fast 1h/5m and slow 6h/30m.
• Dashboards as code and structured JSON logs correlated with metrics in Grafana/Loki.
• Alert quality — grouping + inhibition (don't page for latency when the service is already down).
• Failure injection as proof — scripted scenarios and a place to keep the screenshots.
• Reproducible tooling — a Nix flake with one dev shell per work area.

Architecture

kind cluster — 1 control-plane + 2 workers
│
├─ namespace: default
│   └─ sample-app ×3   FastAPI · RED metrics at /metrics · JSON logs → stdout
│         │                                   │
│         │ scraped via ServiceMonitor        │ logs
│         ▼                                   ▼
├─ namespace: monitoring                  Promtail (DaemonSet, 1/node)
│   ├─ Prometheus (+ Operator)                │ push
│   │     └─ recording rules → burn-rate alerts → Alertmanager
│   │                                           (routing · grouping · inhibition)
│   ├─ Grafana — dashboards-as-code · datasources: Prometheus + Loki
│   ├─ node-exporter + kube-state-metrics — USE / node + cluster metrics
│   └─ Loki (single-binary) ◄──────────────── Promtail

SLOs & error budget

• Availability — 99.5% of requests non-5xx over 28 days → a 0.5% error budget.
• Latency — p95 request latency under 500ms.

Recording rules precompute the error-ratio SLI over 5m/30m/1h/6h windows plus p95 latency, so alerts and dashboards query cheap, stable series. The burn-rate alerts compare those ratios against multiples of the budget (14.4× fast page, 6× slow page).

Repo layout

observ/
├── flake.nix                  # dev shells, one per area
├── Makefile                   # operator interface (make help)
├── README.md
├── .envrc · .gitignore · .yamllint.yaml · ruff.toml
├── kind/cluster.yaml          # 1 control-plane + 2 workers
├── app/                       # instrumented FastAPI service
│   ├── Dockerfile · .dockerignore · main.py · requirements.txt
│   └── tests/test_app.py
├── helm/sample-app/           # chart: deployment, service, servicemonitor
├── observability/
│   ├── kube-prometheus-stack.values.yaml · loki.values.yaml · promtail.values.yaml
│   ├── prometheus/            # recording-rules · slo-burn-rate-alerts · tests/
│   ├── alertmanager/          # routing + inhibition (values overlay)
│   └── grafana/dashboards/    # slo-overview · service-golden-signals
├── slo/slo-definitions.yaml
├── load/                      # steady.js (k6) · failure-scenarios.md
├── runbooks/                  # high-error-rate · high-latency-p95 · service-down
├── docs/evidence/             # screenshots from the proof run
└── .github/workflows/ci.yaml

Prerequisites

Nix (with flakes) and Docker. Everything else (kind, kubectl, helm, k6, promtool, …) is pinned by the flake.

Before anything: replace CHANGEME with your GitHub owner in flake.nix, helm/sample-app/values.yaml, the alert runbook_urls, and the dashboards.

Quickstart

nix develop .#ops      # tooling for cluster ops
make bootstrap         # nothing -> a running, monitored app

Then, in separate terminals:

make forward-app          # http://localhost:8080  (/healthz /work /metrics)
make forward-prom         # http://localhost:9090  -> Status > Targets (sample-app UP)
make forward-grafana      # http://localhost:3000  (admin/admin) -> "observ — SLO overview"
make forward-alertmanager # http://localhost:9093

"Working" = querying http_requests_total in Prometheus returns series.

Dev shells

Shell	Area	Key tooling
nix develop .#dev	Development	python (fastapi/uvicorn/prometheus-client/pytest), ruff, docker
nix develop .#ops	Deployment	kind, kubectl, helm, stern, docker
nix develop .#reliability	Observability	promtool, amtool, kubectl, helm
nix develop .#ci	CI/CD & load	gh, act, actionlint, yamllint, k6, promtool
nix develop .#docs	Docs & evidence	k6, kubectl, stern, markdownlint

nix develop with no target prints the picker. With direnv installed, direnv allow auto-enters the shell from .envrc.

Make targets

make help lists them all. Most-used:

• Lifecycle — up, stack, build, load-image, deploy, down, bootstrap
• Rules & dashboards — rules (wraps the plain rule files into PrometheusRules), dashboards
• Validate — test-rules (promtool, no cluster needed), lint
• Proof — load, inject-errors RATE=0.3, inject-latency MS=800, kill-pod
• Access — forward-app, forward-prom, forward-grafana, forward-alertmanager

Proving it works

The artifact that sells the project is the evidence that detection fires. Run steady traffic, then trigger each scenario in load/failure-scenarios.md and screenshot the result into docs/evidence/:

nix develop .#docs
make forward-app &     # in one terminal
make load              # steady 20 rps
make inject-errors RATE=0.3   # -> SampleAppErrorBudgetBurnFast

(images appear once you capture them — see docs/evidence/README.md.)

Testing the alerts

nix develop .#reliability
make test-rules        # promtool unit tests over synthetic series

CI

.github/workflows/ci.yaml runs every gate through the matching dev shell (ruff, pytest, promtool, yamllint, actionlint, helm lint), then builds and pushes the image to GHCR on main.

Gotchas worth knowing

• ServiceMonitor discovery — the Operator only selects ServiceMonitors labelled release=<stack> unless serviceMonitorSelectorNilUsesHelmValues: false (set in the stack values). Miss this and the target never appears.
• kind images — kind load docker-image puts the image on the nodes; pullPolicy: IfNotPresent makes them use it instead of trying to pull from GHCR.
• Loki chart — its values schema shifts between major versions; diff against helm show values grafana/loki.

Status

A learning project. Validate changes with make test-rules, helm lint ./helm/sample-app, and helm template — the rule/metric names are kept consistent across the app, recording rules, alerts, promtool tests, and dashboards.