Mini AI Inference Platform

A small but complete DevOps portfolio project: a FastAPI machine-learning service, packaged as a container, deployed to a multi-node Kubernetes cluster, observed with Prometheus + Grafana, and shipped through a CI/CD pipeline — all reproducible from a single Nix flake on NixOS.

It is deliberately small on the ML side (a tiny sentiment classifier) so the focus stays on the platform: packaging, scheduling, observability, automation, and the reproducible developer environment.

---

Why

Most "deploy an app to Kubernetes" demos skip the parts that actually matter in practice: a reproducible toolchain, sane multi-node scheduling, real metrics and alerts, and a pipeline that proves the chart installs. This project wires all of those together end to end, in a way you can stand up on a laptop in well under an hour and tear down just as fast.

What you get

• Inference service — FastAPI with /predict, /health, and a Prometheus /metrics endpoint. The model (TF-IDF + logistic regression) trains at startup, so there's no artifact to mount.
• Two ways to build the image — a conventional multi-stage Dockerfile, and a reproducible dockerTools image from the flake (nix build .#dockerImage).
• Infrastructure as code — a kind cluster (1 control-plane + 2 workers) provisioned with OpenTofu, with a CLI fallback.
• Helm chart — replicas spread across workers via topologySpreadConstraints, health probes, an optional node-pin to the labeled inference node, and a gated ServiceMonitor.
• Observability — kube-prometheus-stack, a ServiceMonitor, a 3-panel Grafana dashboard (request rate, p95 latency, error rate), and alert rules (high error rate, app down).
• CI/CD — GitHub Actions: lint + test, build & push to GHCR, and a deploy-validate job that installs the chart on a throwaway kind cluster.
• Automation — a health-check poller wired to a NixOS systemd.timer.
• Reproducible dev env — one flake, with a separate shell per area plus a composed all shell.

How it fits together

Five areas, built in dependency order:

flowchart TD
  dev["1 · Dev<br/>app + image"]
  deploy["2 · Deploy<br/>kind + Helm"]
  obs["3 · Observability<br/>metrics + alerts"]
  cicd["4 · CI/CD<br/>pipeline + cron"]
  docs["5 · Docs<br/>runbook + evidence"]

  dev --> deploy
  deploy --> obs
  deploy --> cicd
  obs --> docs
  cicd --> docs
  dev -. "/metrics" .-> obs
  dev -. "Dockerfile + tests" .-> cicd

Loading

At runtime, requests hit a ClusterIP Service that load-balances across replicas spread over the worker nodes; Prometheus scrapes /metrics and feeds Grafana and Alertmanager:

flowchart LR
  client([client]) -->|/predict| svc["Service: app"]
  subgraph kind["kind cluster — 1 control-plane + 2 workers"]
    svc --> p1["pod: app"]
    svc --> p2["pod: app"]
    prom["Prometheus"] -->|scrape /metrics| svc
    prom --> graf["Grafana"]
    prom --> am["Alertmanager"]
  end

Loading

---

Repo layout

.
├── flake.nix                 # per-area dev shells + composed `all` + nix image
├── .envrc                    # direnv: `use flake`
├── Makefile                  # area-grouped tasks (run `make help`)
├── pyproject.toml
├── Dockerfile / .dockerignore
├── app/                      # FastAPI service (main.py, model.py)
├── tests/                    # pytest
├── kind/kind-config.yaml     # cluster topology
├── infra/                    # OpenTofu (kind provider)
├── chart/                    # Helm chart for the app
├── observability/            # kube-prometheus-stack values, alerts, dashboard
├── automation/health_poll.sh # health poller
├── scripts/predict.sh        # convenience client
├── nixos/                    # NixOS modules (workstation foundation + timer)
├── .github/workflows/ci.yml  # CI/CD
├── demo.tape                 # vhs recording script for the demo GIF
├── BUILD.md                  # build checklists per area
└── CHEATSHEET.md             # every command + gotchas + incident runbook

---

Quickstart

Prerequisites: NixOS (or Nix with flakes enabled) and Docker. On NixOS, import nixos/configuration-snippet.nix and nixos-rebuild switch first — it enables Docker, nix-ld (needed for OpenTofu provider binaries), and raises the inotify limits Kubernetes needs.

# 1. enter the composed dev shell (all tooling)
direnv allow        # or: nix develop

# 2. test the service
make dev-test

# 3. build the image
make image-build

# 4. stand up the whole stack (cluster -> monitoring -> deploy -> alerts/dashboard)
make all-up

# 5. use it
make app-port                      # in one terminal
curl localhost:8000/health         # in another
./scripts/predict.sh "i love this"

# 6. tear it down
make down

Run make help to see every target, grouped by area. CHEATSHEET.md has the full command reference plus the gotchas; BUILD.md is the step-by-step checklist.

---

The Nix dev shells

The flake exposes one shell per area plus a composed shell. Each carries its own tooling and its own environment variables (cluster name, ports, image name, namespaces, …):

Command	Area	Tooling
nix develop .#dev	1 · Dev	python, uv, ruff, docker, hadolint
nix develop .#deploy	2 · Deploy	kubectl, kind, helm, opentofu, kubectx, k9s
nix develop .#observe	3 · Observability	kubectl, helm, k9s, yq
nix develop .#cicd	4 · CI/CD	act, gh, kubectl, kind, helm, shellcheck
nix develop .#docs	5 · Docs	vhs, glow
nix develop (default = all)	everything	composed via inputsFrom

The all shell is built with inputsFrom = [ dev deploy observe cicd docs ], so it inherits every area's packages; its environment is the merge of all per-area env sets. (inputsFrom propagates packages but not env vars, which is why the env lives in named attrsets that the combined shell merges explicitly.)

---

Observability

make monitoring installs kube-prometheus-stack; the chart's ServiceMonitor is auto-discovered (the stack is configured with serviceMonitorSelectorNilUsesHelmValues: false). make dashboard imports the Grafana dashboard via a labeled ConfigMap, and make alerts applies the PrometheusRule. Reach the UIs with make grafana / make prometheus / make alertmanager. To see an alert fire: make fire-alert (scales the app to 0), then make heal.

CI/CD — honest CD

The pipeline lints, tests, and builds & pushes the image to GHCR on main. Cloud runners can't reach your local cluster, so "CD" here means proving the chart deploys: the deploy-validate job spins up a throwaway multi-node kind cluster, loads a locally built image, helm installs the chart, and smoke-tests /health from inside the cluster.

Stretch goals

• Two-cluster GitOps (hub/spoke) — register the spoke by its Docker-network IP, not 127.0.0.1 (see the gotchas in CHEATSHEET.md).
• Loki for logs alongside the metrics stack.
• An on-prem VM track via libvirt (NixOS module stubs are included, commented out).

See BUILD.md for the checklist and CHEATSHEET.md for the incident runbook.