Repository navigation
NInfer V100 v1.5.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
Highlights
Decode is 13–28% faster than v1.3.0, from decode-side kernel work, a deeper speculative draft and a rewritten PV epilogue in the INT8 attention kernel.
Qwen3.8-27B nvfp4, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB, 1024 generated tokens per run.
| Prompt tokens | Prompt | v1.3.0 (draft 3) | v1.5.0 (draft 4) | Change |
|---|---|---|---|---|
| 7,353 | code | 81.1 | 101.1 | +25% |
| 7,792 | Chinese | 59.2 | 67.1 | +13% |
| 31,538 | code | 74.3 | 94.8 | +28% |
| 31,924 | Chinese | 54.8 | 57.5 | +5% |
| 186,420 | code | 51.3 | 62.2 | +21% |
| 193,104 | Chinese | 38.1 | 39.9 | +5% |
Numbers are decode tok/s. First token at 186K is unchanged from v1.3.0 (341 s, 547 tok/s cold prefill).
Round time is the drift-free quantity; the same runs give (code prompts):
| round time | 8K | 32K | 186K |
|---|---|---|---|
| v1.3.0, draft 3 | 39.5 ms | 43.0 ms | 63.0 ms |
| v1.4.0, draft 4 | 38.6 ms | 41.3 ms | 62.5 ms |
| v1.5.0, draft 4 | 38.3 ms | 41.0 ms | 60.0 ms |
What changed since v1.3.0
- Three decode-side ports from the same source: fp16 activation staging for the QPN residual and down projections (the kernel was converting every activation load), the NVFP4 residual add fused into the QPN epilogue, and a wider small-T split reduce that keeps several loads in flight per output element.
- Chained fp16 PV accumulator in the INT8 small-T decode attention kernel: one accumulator per output chunk across the whole 64-key tile instead of a fresh accumulator and a float conversion per 8-key group. An ablation build shows that epilogue was 26% of the kernel while the tile loads were 8%, i.e. the kernel is instruction-issue bound rather than bandwidth bound.
NINFER_SM70_ATTN_V3=0restores the previous epilogue for in-image A/B;NINFER_SM70_ATTN_V2=0still falls back to the original 4-warp kernel. - Draft depth: the deployment default moves to
--draft-tokens 4, worth +9% to +17% over draft 3 at 8K–32K and +14% at 186K. Draft 5 is slower above 32K (118.8 ms per round at 186K). - Opt-in draft key window:
NINFER_MTP_ATTN_WINDOW=<keys>lets the MTP layer attend to the last N keys only (+11% at 186K code, −7% at 32K, off by default). With speculation disabled it produces byte-identical output to window 0, so target-model attention is provably untouched. - Documentation fix, no behaviour change: the long-context prefill cost was misattributed to the KV materialization planner in an earlier draft of the plan document. Instrumented counters show the engine's per-chunk helpers are ~0 and the 149 s host part of the 337 s 186K prefill is eager per-layer submission inside the model's prefill execution. It is documented, not fixed, in this release.
Verification
ninfer_softmax_attention_test(public-contract oracle comparison) passes withNINFER_SM70_ATTN_V3both on and off;ninfer_qwen3_6_runtime_mechanisms_testpasses; the DFlash2 real-model integration test accepts 20/20 atK=7 B=2.- Every number above is paired with a repeated baseline in the same session; the drift control matched its first run exactly (8K 38.7/38.7 ms, 32K 41.5/41.5 ms, 186K 62.9/62.9 ms).
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.5.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license |
ninfer-serve-sm70.tar.gz |
server binary only, ready for docker/README.md step 1 |
SHA256SUMS.txt |
checksums |
Docker
docker pull ghcr.io/jimmymax/ninfer-v100:v1.5.0See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table. For measurement protocol and the V100 performance plan see bench/v100/README.md and docs/V100-PERF-PLAN.md.
Full changelog: v1.3.0...v1.5.0