You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cloud-Scale Multi-Format ML Model Serving Platform
A highly resilient, long-term, multi-format ML model serving platform with triple-layer separation, designed to serve millions of users with auto-repair, capacity forecasting, and multi-year durability.
The platform is built on a triple-layer separation principle that ensures maximum modularity and long-term maintainability. The key insight: never rigidly couple the model format to the serving engine.
The Three Planes
graph TB
subgraph Exposure["EXPOSURE PLANE (Uniform API)"]
GW["Envoy AI Gateway<br/>OpenAI-Compatible API (/v1/chat/completions)"]
GW_FEAT["HTTPRoute · Auth (APIKey) · Rate Limiting<br/>Cost Metrics · SSE Streaming"]
GW_RESIL["Priority Routing (0to1) · Circuit Breaker<br/>Retry (502/503/504) · Failover (>2000ms)"]
GW --- GW_FEAT
GW --- GW_RESIL
end
subgraph Engine["ENGINE PLANE (vLLM Only)"]
VLLM["vLLM<br/>port 8000"]
end
subgraph Model["MODEL PLANE (Interchangeable Weights)"]
SAFE["Safetensors<br/>BF16/FP16"]
AWQ["AWQ/GPTQ"]
end
subgraph Storage["Storage"]
MINIO["MinIO<br/>(S3-compatible)"]
PVC["RWX PVC<br/>(Longhorn)"]
MOUNT["/models/<name>/"]
MINIO -->|seed| PVC
PVC -->|mount| MOUNT
end
Exposure ==> Engine
Engine ==> Model
Model -.->|weights loaded from| Storage
Loading
Why this matters:
Add a new model → no gateway change needed
Change engine → no client-side change needed
Switch to SaaS fallback → transparent to end users
Each plane evolves independently over years
High-Level Architecture Diagram
graph TB
GIT["Git Repository<br/>charts/ environments/ models/ apps/<br/>tools/ observability/ tests/ docs/"]
CI["GitHub Actions CI<br/>Rust build+test · Helm lint (3)<br/>Registry check · VRAM validation"]
ARGOCD["ArgoCD Control (Control Cluster)<br/>ApplicationSets: model-serving, ai-gateway,<br/>infrastructure (GPU/LH/Prom), secrets (ESO)<br/>Custom Lua Health: StatefulSet, InferenceService"]
GIT -->|push| CI
CI -->|if pass| ARGOCD
ARGOCD -->|Sync Waves -3 to 2| WA
ARGOCD -->|Sync Waves -3 to 2| WB
ARGOCD -->|Sync Waves -3 to 2| WDEV
subgraph WA["Worker Cluster (Region A)"]
WA_WAVES["Waves: -3 Secrets to -2 Longhorn PVC to -1 GPU Operator<br/>to 0 Model Pods to 1 Gateway+Dashboards to 2 Smoke Tests"]
WA_POOLS["Node Pools: gpu-h100-pool · gpu-a100-pool<br/>gpu-l4-pool · cpu-pool"]
end
subgraph WB["Worker Cluster (Region B)"]
WB_WAVES["Waves: -3 Secrets to -2 Longhorn PVC to -1 GPU Operator<br/>to 0 Model Pods to 1 Gateway+Dashboards to 2 Smoke Tests"]
WB_POOLS["Node Pools: gpu-h100-pool · gpu-a100-pool<br/>gpu-l4-pool · cpu-pool"]
end
subgraph WDEV["Worker Cluster (Edge / Dev)"]
WDEV_WAVES["Waves: -3 Secrets to -2 local-path to -1 GPU Operator<br/>to 0 Model Pods to 1 Gateway+Dashboards to 2 Smoke Tests"]
WDEV_POOLS["Node Pools: gpu-edge-pool (A2000)<br/>gpu-l4-pool (L4) · cpu-pool"]
end
Loading
Request Flow Diagram
sequenceDiagram
participant C as "Client (SDK / curl)"
participant GW as "Envoy AI Gateway"
participant CB as "Circuit Breaker (Prioritized)"
participant V as "vLLM port 8000"
participant S as "models PVC (RWX via Longhorn)"
C->>GW: POST /v1/chat/completions<br/>Authorization: Bearer key
GW->>CB: HTTPRoute to BackendTrafficPolicy
CB->>CB: priority 0 to 1, retry 502/503/504
CB->>CB: Health Check GET /health (10s interval)
CB->>CB: Failover if latency gt 2000ms, priority 1 (SaaS fallback)
CB->>V: route to vLLM
V->>S: load weights
V-->>GW: SSE stream
GW-->>C: SSE stream choices
# Build all tools in the workspace
cargo build --release
# Or build individually
cargo build --release --bin engine-selector
cargo build --release --bin vram-budget-calc
cargo build --release --bin model-onboarding
cargo build --release --bin cache-roi-calc
2. Run Tests
# Run all unit tests (61 tests across 4 crates)
cargo test# Run tests for a specific tool
cargo test --bin engine-selector # 31 tests
cargo test --bin vram-budget-calc # 16 tests
cargo test --bin model-onboarding # 14 tests
cargo test --bin cache-roi-calc # 0 tests (CLI tool, no unit tests)
3. Use the Tools
# Select the best engine for a model
./target/release/engine-selector --model /path/to/model --json
# Override format detection
./target/release/engine-selector --model /path/to/model --format awq
# Calculate VRAM budget
./target/release/vram-budget-calc \
--total-vram 8 \
--model-size 4.7 \
--quant awq \
--gpu "RTX A2000" \
--batch 1 \
--context 8192 \
--layers 32 \
--heads 32 \
--json
# Onboard a new model (scaffolds files)
./target/release/model-onboarding \
--name my-model \
--format safetensors \
--vram-budget 8 \
--gpu-pool "RTX A2000" \
--dry-run
GPU health (temperature, utilization, memory, ECC)
model-serving-dashboard.json
Request rate, P95 latency, error rate, tokens/s, OOM kills, KV cache usage (%), prefix cache hit rate (%), request queue depth, TTFT (p95+p50), KV cache swap-out blocks, GPU VRAM usage (DCGM), LMCache L1/L2/L3 hit rates, prefill skip rate, cache ROI estimate ($/h), cache affinity routing distribution — 18 panels total
Alerting Rules
Category
Alert
Condition
Severity
Latency
HighLatency
p95 > 2s for 3m
Warning
Latency
CriticalLatency
p99 > 5s for 2m
Critical
Errors
HighErrorRate
> 5% for 5m
Warning
Errors
CriticalErrorRate
> 15% for 3m
Critical
GPU
GPUThermalThrottle
> 85°C
Critical
GPU
GPUUtilizationLow
< 10% for 30m
Warning
GPU
GPUMemoryNearExhaustion
> 95% for 2m
Critical
GPU
GPUEccErrors
> 100/h
Critical
Pods
CrashLooping
restarts > 3/h
Warning
Pods
NotReady
10m
Warning
Anomaly
LatencyAnomaly
deriv > 0.1 for 10m
Warning
Anomaly
ThroughputAnomaly
deriv < -0.5 for 10m
Warning
KV Cache
VLLMKVCacheUsageHigh
vllm:gpu_cache_usage_perc > 0.85 for 30s
Warning
KV Cache
VLLMKVCacheUsageCritical
vllm:gpu_cache_usage_perc >= 1.0
Critical
KV Cache
VLLMRequestsWaitingHigh
vllm:num_requests_waiting > 10 for 1m
Critical
KV Cache
VLLMSwapOutBlocksDetected
increase(vllm:swap_out_blocks[5m]) > 0
Critical
KV Cache
NodeSwapSpaceUsageHigh
swap usage > 10% for 2m
Critical
KV Cache
VLLMPrefixCacheHitRateLow
prefix cache hit < 20% for 10m
Warning
KV Cache
LMCacheL1HitRateLow
L1 CPU hit < 30% for 10m
Warning
KV Cache
LMCacheL2HitRateLow
L2 NVMe hit < 20% for 15m
Warning
KV Cache
LMCacheL3HitRateLow
L3 Redis/S3 hit < 10% for 15m
Warning
KV Cache
SSMModelPagedAttentionMisconfigured
SSM pod with PagedAttention args
Critical
KV Cache
VLLMPrefillSkipRateLow
prefill skip < 10% while queue busy
Info
KV Cache
CacheRoutingHeaderAbsent
x-cache-affinity-key missing during traffic
Info
Alert Routing
Critical → PagerDuty + Slack #ml-incidents
Warning → Slack #ml-ops
GPU → Slack #gpu-ops
Serving → Slack #ml-ops
Inhibit: critical suppresses warning for same alert
KV Cache Management
The platform implements an 8-layer defensive architecture for vLLM KV cache management, as documented in docs/explain/kv-cache.md and docs/explain/bible-kv-cache.md. Each layer protects the KV cache from a different failure mode.
Layer 1 — API Gateway (Edge Protection)
Mechanism
Implementation
Failure Mode Prevented
Payload validation
HTTPRouteFilter maxBodySize 4MiB → HTTP 413
Oversized payloads polluting KV cache
Rate limiting
BackendTrafficPolicy 50 req/s per x-api-key → HTTP 429
Request floods overwhelming KV cache
Sticky routing
x-sticky-session-key header → same replica
Prefix cache misses from random routing
Aggressive timeouts
request 10s / backendRequest 8s
Queue thrashing from slow requests
Layer 2 — vLLM Engine (Cache Efficiency)
Argument
Prod
Staging
Dev
Purpose
--gpu-memory-utilization
0.90
0.88
0.85
Reserve headroom for KV cache growth
--max-model-len
8192
8192
4096
Cap context length to business need
--max-num-seqs
256
128
64
Limit concurrent sequences in KV cache
--kv-cache-dtype
fp8
fp8
fp8
Halve KV cache memory via quantization
--enable-prefix-caching
✓
✓
✓
Reuse KV cache for shared prefixes
--block-size
16
16
16
Optimal block size for paged attention
--tensor-parallel-size
1
1
1
Per-NVLink topology
Layer 3 — Kubernetes (Resource Protection)
Mechanism
Implementation
Failure Mode Prevented
QoS Guaranteed
requests == limits (CPU/RAM/GPU) in all envs
Host OOM killer evicting vLLM pods
swapoff DaemonSet
nsenter swapoff -a on GPU nodes via DaemonSet
Host swapping KV cache pages to CPU RAM
Node isolation
nodeSelector: nvidia.com/gpu.present: "true"
CPU workloads competing for GPU node RAM
Layer 4 — Observability (Early Detection)
Alert
Condition
Severity
VLLMKVCacheUsageHigh
vllm:gpu_cache_usage_perc > 0.85 for 30s
Warning
VLLMKVCacheUsageCritical
vllm:gpu_cache_usage_perc >= 1.0
Critical
VLLMRequestsWaitingHigh
vllm:num_requests_waiting > 10 for 1m
Critical
VLLMSwapOutBlocksDetected
increase(vllm:swap_out_blocks[5m]) > 0
Critical
NodeSwapSpaceUsageHigh
swap usage > 10% for 2m
Critical
VLLMPrefixCacheHitRateLow
hit rate < 20% for 10m
Warning
LMCacheL1HitRateLow
L1 CPU hit < 30% for 10m
Warning
LMCacheL2HitRateLow
L2 NVMe hit < 20% for 15m
Warning
LMCacheL3HitRateLow
L3 distributed hit < 10% for 15m
Warning
VLLMPrefillSkipRateLow
prefill skip < 10% while queue busy
Info
SSMModelPagedAttentionMisconfigured
SSM pod detected with PagedAttention args
Critical
CacheRoutingHeaderAbsent
x-cache-affinity-key missing during traffic
Info
ServiceMonitor scrapes vLLM /metrics every 10s with honorLabels: true. Grafana dashboard has 18 panels including 6 LMCache hierarchy and ROI panels (see docs/architecture/05-observability.md).
Layer 5 — Autoscaling (KEDA)
Classic CPU/RAM HPA is inoperant for LLM workloads (GPU-bound, not CPU-bound). The platform uses a KEDA ScaledObject with two Prometheus triggers:
Trigger
Metric
Threshold
Action
Queue depth
vllm:num_requests_waiting
> 5
Scale out
Cache pressure
vllm:gpu_cache_usage_perc
> 0.85
Scale out
minReplicaCount: 2 (prod), maxReplicaCount: 4
pollingInterval: 15s, cooldownPeriod: 60s
Legacy HPA fallback retained for environments without KEDA
Layer 6 — GitOps (Change Safety)
All critical vLLM params centralized in environments/{dev,staging,prod}/values.yaml
ArgoCD sync waves with self-heal + prune + ServerSideApply
vram-budget-calc CI gate blocks deployment if KV cache budget < 0
cache-roi-calc CLI tool computes ROI ratio, GPU savings, and break-even hit rate (Bible §9)
k6 load tests validate before changes reach production
Staging environment uses identical GPU hardware to prod
Layer 7 — Distributed Cache Middleware (LMCache)
The platform deploys LMCache as a per-GPU-node DaemonSet to break the per-instance KV cache silo (Bible §4.3). Cache becomes shareable across pods, persistent across restarts, and hierarchical across memory tiers.
SafeTensors Cache Persistence — cachePersistence provisions a dedicated PVC (/cache/kv) via Longhorn so the KV cache survives pod restarts. On startup, vLLM restores from the persisted cache, reducing TTFT from ~11s to ~1.5s on a 128K context at 80% hit rate.
Sticky routing by model name (x-sticky-session-key header) is augmented with consistent-hash load balancing on the x-cache-affinity-key header, derived from a prefix hash of the first 64 tokens of the system prompt.
Mechanism
Implementation
Failure Mode Prevented
Prefix-hash routing
Envoy Lua filter (FNV-1a over first 512 bytes of body) → x-cache-affinity-key header
Random routing destroying prefix cache locality
Consistent-hash LB
BackendTrafficPolicyloadBalancer.type: ConsistentHash on x-cache-affinity-key
Template: cache-routing-policy.yaml (ConfigMap with Lua filter + invalidation policy).
Multi-Family Model Support — The engine-selector tool detects model family (Transformer, MoE, SSM/Mamba, Hybrid) and returns the correct cache strategy. SSM/Mamba models use a fixed-size recurrent state — --enable-prefix-caching and --block-size are misconfigurations for them (Bible §14). The SSMModelPagedAttentionMisconfigured alert catches this at runtime.
CI/CD Pipeline
The GitHub Actions workflow (.github/workflows/ci.yaml) runs 4 jobs:
Job
Description
Blocking
rust-tools
Build + test all 4 crates, clippy (deny warnings), fmt check
Yes
helm-lint
Lint all 3 charts + template dry-run with test values
Yes
registry-consistency
Validate each registry entry has chart dir, model dir, and required files
Yes
vram-budget-validation
Build vram-budget-calc and run for all LIVE/STAGED models — fails if budget exceeded
The platform integrates with external systems via GitOps. All integration manifests live in apps/ and addons/. See docs/integration-report.md for the complete 13-section report.
