dispatch v1.0.0 -- inference engine for MoE token routing
Full 8-phase system design complete, Phase 0 through Phase 7: a from-scratch, correctness-proven Triton grouped-GEMM kernel; real multi-GPU expert-parallel serving over DeepSeek's own DeepEP; disaggregated prefill/decode across two real 4-GPU topologies; self-computed int8 weight-only quantization; speculative decoding with two drafters; a head-to-head benchmark against vLLM and SGLang on identical hardware, reported as the real loss and real win it was, not reframed; and a Rust router in front of a real GPU-backed model server, containerized and deployed to Kubernetes, demonstrated against a rented GPU.
An open-source contribution landed along the way: an opt-in skewed-load benchmark flag upstreamed to vLLM (vllm-project/vllm#57100).
Every benchmark number in this project was measured on real, rented GPU hardware -- never estimated. Every null and mixed result is reported as such rather than smoothed over.
Total measured GPU cost across all eleven sessions: $51.35.
Per-phase detail
Each phase has its own git tag and a full findings write-up:
| Phase | Tag | Result | Cost |
|---|---|---|---|
| 0 -- Baseline | phase-0 |
12.75 tok/s, 0.355s mean TTFT | $0.35 |
| 1 -- Triton grouped-GEMM kernel | phase-1 |
+65-67% over stock, perfect logit agreement | $0.65 |
| 2 -- vLLM benchmark contribution | phase-2 |
Upstreamed PR, open | $0.77 |
| 3 -- Multi-GPU expert-parallel (DeepEP) | phase-3 |
Byte-exact; disproved own crossover hypothesis | $13.03 |
| 4 -- Disaggregated prefill/decode | phase-4 |
Mixed result, reported as inconclusive | $10.94 |
| 5a -- int8 quantization | phase-5a |
49.89% memory reduction, perfect agreement | $1.95 |
| 5b -- Speculative decoding | phase-5b |
Prompt-lookup beats baseline; RoPE bug root-caused | $12.26 |
| 6 -- Final benchmark vs. vLLM/SGLang | phase-6 |
vLLM wins every shape, disclosed loss | $4.5448 |
| 7 -- Productionization | phase-7 |
Rust router + K8s, real tokenizer bug fixed | $6.8563 |
Full write-ups: docs/findings/. Phase-by-phase changelog: CHANGELOG.md.