Skip to content

NInfer V100 v1.5.0

Choose a tag to compare

@github-actions github-actions released this 11 Oct 12:50

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

Highlights

Decode is 13–28% faster than v1.3.0, from decode-side kernel work, a deeper speculative draft and a rewritten PV epilogue in the INT8 attention kernel.

Qwen3.8-27B nvfp4, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB, 1024 generated tokens per run.

Prompt tokens Prompt v1.3.0 (draft 3) v1.5.0 (draft 4) Change
7,353 code 81.1 101.1 +25%
7,792 Chinese 59.2 67.1 +13%
31,538 code 74.3 94.8 +28%
31,924 Chinese 54.8 57.5 +5%
186,420 code 51.3 62.2 +21%
193,104 Chinese 38.1 39.9 +5%

Numbers are decode tok/s. First token at 186K is unchanged from v1.3.0 (341 s, 547 tok/s cold prefill).

Round time is the drift-free quantity; the same runs give (code prompts):

round time 8K 32K 186K
v1.3.0, draft 3 39.5 ms 43.0 ms 63.0 ms
v1.4.0, draft 4 38.6 ms 41.3 ms 62.5 ms
v1.5.0, draft 4 38.3 ms 41.0 ms 60.0 ms

What changed since v1.3.0

  • Three decode-side ports from the same source: fp16 activation staging for the QPN residual and down projections (the kernel was converting every activation load), the NVFP4 residual add fused into the QPN epilogue, and a wider small-T split reduce that keeps several loads in flight per output element.
  • Chained fp16 PV accumulator in the INT8 small-T decode attention kernel: one accumulator per output chunk across the whole 64-key tile instead of a fresh accumulator and a float conversion per 8-key group. An ablation build shows that epilogue was 26% of the kernel while the tile loads were 8%, i.e. the kernel is instruction-issue bound rather than bandwidth bound. NINFER_SM70_ATTN_V3=0 restores the previous epilogue for in-image A/B; NINFER_SM70_ATTN_V2=0 still falls back to the original 4-warp kernel.
  • Draft depth: the deployment default moves to --draft-tokens 4, worth +9% to +17% over draft 3 at 8K–32K and +14% at 186K. Draft 5 is slower above 32K (118.8 ms per round at 186K).
  • Opt-in draft key window: NINFER_MTP_ATTN_WINDOW=<keys> lets the MTP layer attend to the last N keys only (+11% at 186K code, −7% at 32K, off by default). With speculation disabled it produces byte-identical output to window 0, so target-model attention is provably untouched.
  • Documentation fix, no behaviour change: the long-context prefill cost was misattributed to the KV materialization planner in an earlier draft of the plan document. Instrumented counters show the engine's per-chunk helpers are ~0 and the 149 s host part of the 337 s 186K prefill is eager per-layer submission inside the model's prefill execution. It is documented, not fixed, in this release.

Verification

  • ninfer_softmax_attention_test (public-contract oracle comparison) passes with NINFER_SM70_ATTN_V3 both on and off; ninfer_qwen3_6_runtime_mechanisms_test passes; the DFlash2 real-model integration test accepts 20/20 at K=7 B=2.
  • Every number above is paired with a repeated baseline in the same session; the drift control matched its first run exactly (8K 38.7/38.7 ms, 32K 41.5/41.5 ms, 186K 62.9/62.9 ms).

Assets

File Contents
ninfer-v100-v1.5.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license
ninfer-serve-sm70.tar.gz server binary only, ready for docker/README.md step 1
SHA256SUMS.txt checksums

Docker

docker pull ghcr.io/jimmymax/ninfer-v100:v1.5.0

See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table. For measurement protocol and the V100 performance plan see bench/v100/README.md and docs/V100-PERF-PLAN.md.

Full changelog: v1.3.0...v1.5.0