Fault-tolerant AV perception pipeline with LLM-augmented fallback, SLO monitoring, chaos engineering, and full observability. Built to demonstrate SRE/infra thinking for safety-critical autonomous vehicle systems.
┌─────────────────────────────────────────────┐
│ Modal (T4 GPU) │
│ │
│ carla_server ──► perception/pipeline │
│ (dataset replay) YOLOv8n @ 20 Hz │
│ /ws/detections │
│ /metrics (Prometheus) │
└──────────────────────────┬──────────────────┘
│ wss://
▼
┌──────────────────────────────────────────────┐
│ Local services │
│ │
│ chaos/proxy (:9100) ◄── optional fault │
│ │ ws:// injection │
│ ▼ │
│ watchdog/monitor (:9101) │
│ • rolling p99 tracking │
│ • streak >= 3 breaches -> POST /trigger │
│ │ │
│ ▼ │
│ fallback/server (:8001) │
│ • Nemotron 550B via NVIDIA NIM │
│ • NeMo Guardrails (2-layer validation) │
│ • POST /control on pass │
│ │ │
│ ▼ │
│ control/server (:8002) │
│ • fallback / primary arbiter │
│ • EMA smoothing + rate limits │
│ • vehicle physics sim @ 20 Hz │
│ • /ws/telemetry │
└──────────────────────────────────────────────┘
│
┌──────────────────────────▼──────────────────┐
│ Observability (Docker) │
│ Prometheus (:9090) <- scrapes all 5 jobs │
│ Grafana (:3000) <- SLO dashboard │
└──────────────────────────────────────────────┘
| Metric | Target |
|---|---|
| YOLOv8n p99 inference latency | <= 40 ms |
| Fallback activation threshold | 3 consecutive SLO breaches |
| Fallback pipeline latency (trigger to control) | <= 10 s |
| Guardrails: max throttle in fallback mode | <= 0.5 |
| Guardrails: emergency_stop required above | p99 > 150 ms |
| Layer | Technology |
|---|---|
| Sensor simulation | CARLA 0.9.15 (dataset replay on Modal Volume) |
| Primary perception | YOLOv8n (ultralytics) on Modal T4 GPU |
| Fallback LLM | Nemotron Mini via NVIDIA NIM (nvidia/nemotron-mini-4b-instruct) |
| Output validation | NeMo Guardrails (deterministic + Colang output rails) |
| SLO monitoring | Custom watchdog — rolling p99, breach streak tracking |
| Fault injection | Chaos proxy — 5 fault modes, HTTP-controlled |
| Vehicle control | FastAPI arbiter, EMA smoothing, bicycle-model sim |
| Metrics | Prometheus + Grafana (5 scrape targets, 15+ panels) |
| Cloud compute | Modal (serverless GPU, ASGI deploy) |
export NVIDIA_API_KEY=<your-nvidia-nim-key>
export MODAL_TOKEN_ID=<your-modal-token-id>
export MODAL_TOKEN_SECRET=<your-modal-token-secret># One-time: generate synthetic CARLA dataset
modal run carla_server/server.py::generate_dataset
# Deploy both services and capture URLs
modal deploy carla_server/server.py # -> CARLA_URL
modal deploy perception/pipeline.py # -> PERCEPTION_URLpip install fastapi uvicorn websockets httpx prometheus-client \
nemoguardrails pydantic# PERCEPTION_URL is the https:// serve URL from modal deploy output
# e.g. https://youruser--perception-sentinel-perception-serve.modal.run
WS_URL="wss://$(echo $PERCEPTION_URL | sed 's|https://||')/ws/detections"
# Terminal 1 — watchdog
PERCEPTION_WS_URL=$WS_URL python -m watchdog.monitor
# Terminal 2 — fallback
python -m fallback.server
# Terminal 3 — control
python -m control.server
# Terminal 4 — chaos proxy (optional; redirect watchdog here for fault injection)
UPSTREAM_WS_URL=$WS_URL python -m chaos.proxycd infra && docker compose up -d
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000 (admin / sentinel)Wire watchdog through the chaos proxy to inject faults end-to-end:
# Start chaos proxy
UPSTREAM_WS_URL=wss://$PERCEPTION_URL/ws/detections python -m chaos.proxy
# Point watchdog at proxy instead of perception directly
PERCEPTION_WS_URL=ws://localhost:9100/ws/detections python -m watchdog.monitor| Fault | intensity |
Effect |
|---|---|---|
blackout |
ignored | Drops all frames for duration_s |
frame_drop |
0.0-1.0 (fraction) | Drops that fraction of frames randomly |
latency_spike |
ms to add | Adds Xms delay to every forwarded frame |
corrupt_frame |
0.0-1.0 (fraction) | Replaces that fraction with invalid JSON |
p99_inflation |
ms above SLO | Rewrites p99_ms = 40 + intensity, slo_ok = False |
# Experiment 1: SLO breach -> Nemotron fallback (conservative)
# intensity=40 → p99 reported as 80ms; watchdog fires after 3 consecutive breaches
curl -X POST localhost:9100/inject \
-H "Content-Type: application/json" \
-d '{"fault_type":"p99_inflation","intensity":40,"duration_s":30}'
# Expected log sequence:
# [watchdog] BREACH p99=80.0ms streak=1/3
# [watchdog] BREACH p99=80.0ms streak=2/3
# [watchdog] BREACH p99=80.0ms streak=3/3
# [watchdog] threshold reached — firing fallback
# [fallback] Nemotron -> conservative_follow (conf=0.87)
# [fallback] done action=conservative_follow guardrails=ok
# [control] fallback conservative_follow thr=0.25 brk=0.00
# Experiment 2: Severe breach -> emergency_stop (guardrails override)
# intensity=120 → p99 reported as 160ms; deterministic guardrail forces emergency_stop
curl -X POST localhost:9100/inject \
-H "Content-Type: application/json" \
-d '{"fault_type":"p99_inflation","intensity":120,"duration_s":20}'
# [fallback] guardrails BLOCKED: p99=160ms requires emergency_stop
# [control] fallback emergency_stop thr=0.00 brk=1.00
# Experiment 3: Sensor blackout (complete frame loss)
curl -X POST localhost:9100/inject \
-H "Content-Type: application/json" \
-d '{"fault_type":"blackout","intensity":1,"duration_s":10}'
# Experiment 4: 60% frame drop
curl -X POST localhost:9100/inject \
-H "Content-Type: application/json" \
-d '{"fault_type":"frame_drop","intensity":0.6,"duration_s":60}'
# Experiment 5: 80ms latency spike per frame
curl -X POST localhost:9100/inject \
-H "Content-Type: application/json" \
-d '{"fault_type":"latency_spike","intensity":80,"duration_s":30}'
# Stop any active fault
curl -X DELETE localhost:9100/injectPrometheus scrapes 5 jobs:
| Job | Endpoint | Key metrics |
|---|---|---|
perception |
Modal HTTPS | perception_p99_latency_ms, perception_slo_breach_total |
watchdog |
:9101/metrics |
watchdog_consecutive_breach_count, watchdog_fallback_trigger_total |
fallback |
:8001/metrics |
fallback_guardrails_block_total, fallback_pipeline_latency_seconds |
control |
:8002/metrics |
control_vehicle_speed_ms, control_fallback_active |
chaos |
:9100/metrics |
chaos_frames_dropped_total, chaos_fault_active |
The Grafana dashboard (observability/grafana_dashboard.json) has 15 panels:
p99 time series with 40ms SLO line, current p99 gauge, breach rate, fps,
inference latency heatmap, fallback trigger counts, guardrails block vs forward,
fallback pipeline latency p50/p99, vehicle speed, active controller
(PRIMARY/FALLBACK), throttle/brake/steering overlay, arbiter switch count.
# Guardrails unit tests — no services needed, instant:
pytest tests/integration.py -k "TestGuardrailsUnit" -v
# Local services only (no Modal, no LLM):
pytest tests/integration.py -m "not modal and not llm" -v
# Full suite:
CARLA_URL=https://... PERCEPTION_URL=https://... NVIDIA_API_KEY=... \
pytest tests/integration.py -v
# Speed up hold-related tests:
FALLBACK_HOLD_SECS=1 pytest tests/integration.py -v99 tests across 8 classes. 17 guardrails unit tests run offline with zero deps beyond pytest. A session-scoped warmup fixture ensures CARLA, perception, and watchdog are all warm before WS tests begin — avoids false failures during Modal cold starts.
Why YOLOv8n, not a larger model?
YOLOv8n with FP16 on T4 runs at ~12ms p99, well inside the 40ms SLO. The SLO is
intentionally tight to make the breach path demonstrable without pathological
conditions. gpu="T4" and half=True are required — CPU inference runs at ~65ms
p99 and breaches the SLO continuously.
Why rolling p99 window, not instantaneous latency? Single-frame spikes from GC pauses or JIT are noise. The watchdog uses a 1000-frame rolling window (~50s at 20 Hz) matching the SLO contract. Three consecutive breaches trigger fallback, filtering transient spikes.
Why Nemotron 550B as fallback, not a deterministic controller? Demonstrates LLM-augmented safety decision-making under perception degradation. The NeMo Guardrails layer (deterministic + Colang output rails) ensures the LLM cannot produce unsafe control values regardless of what it outputs.
Why a chaos proxy, not injecting faults into the perception code? The proxy tests the entire SLO breach to fallback path without Modal redeployment. It simulates sensor failure as an infrastructure event — the realistic failure mode.
Why EMA smoothing in the control arbiter? Snap steering commands from the LLM would be dangerous in a real vehicle. EMA (alpha=0.25) smooths commands while allowing emergency_stop to bypass smoothing and apply immediately.