Skip to content

phase-7

@bsreecharanreddy bsreecharanreddy tagged this 21 Sep 15:56
This closes the system design's entire 8-phase plan, Phase 0 through 7.
A Rust router (tonic gRPC client, axum HTTP, Prometheus metrics) in
front of the existing Python model server, containerized (Docker
multi-stage builds), deployed to a local kind Kubernetes cluster with
Prometheus/Grafana observability, demonstrated once against a real
rented GPU over an SSH tunnel. Found and fixed two real bugs neither
anticipated by the plan: a sys.path invocation issue, and a genuine
defect in deepseek-moe-16b-base's own shipped tokenizer -- a byte-level
BPE vocabulary wired to a SentencePiece pre-tokenizer/decoder expecting a
marker that appears in zero vocab entries, silently dropping every space
in generated text. Root-caused and fixed for real, verified by
round-trip token-id identity. Real demo request through the full stack:
136ms TTFT, correctly-spaced generated text, real Grafana metrics. Cost:
$6.8563 of a $5 cap, exceeded and disclosed mid-session.

Full account: docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md
Runbook: docs/runbooks/phase-7-productionization.md
Assets 2
Loading