Repository navigation
NInfer V100 v1.6.2
NInfer V100 v1.6.2
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
A performance release for the report in #2: the
1.6 line is now faster than 1.5 everywhere the reporter measured the loss. Nine commits merge
upstream's 148 post-1.6.1 commits and add the V100 route work; measured on one V100-PCIE-32GB
(250 W, ECC on), int8 KV, --prefill-chunk 2048, alternating A/B against 1.6.1.
Speculative decode is faster than 1.5 at every depth
--spec mtp --draft-tokens 4 --lm-head-draft, Swift v2 NVFP4 artifact:
| prompt | 1.5 | 1.6.2 | change | acceptance 1.5 → 1.6.2 |
|---|---|---|---|---|
| 502 | 65.7 | 70.7 | +7.6% | 40.5% → 43.2% |
| 2,942 | 62.8 | 65.4 | +4.1% | 39.2% → 40.1% |
| 27,222 | 53.3 | 59.2 | +11.1% | 32.7% → 36.9% |
A second, native-nvfp4 artifact (orcarouter-Qwen3.8-27B-Uncensored-nvfp4) gains +2.8%, +2.4% and
+4.7% at the same depths, so the win is not specific to a GGUF-derived artifact. Plain decode is at
parity, and prefill is at parity at 27K and 1.0% behind at 66K.
Fixed: std::bad_alloc in prefill with a long --max-context
A configuration with --max-context past 131,072 crashed on its first long prefill:
ERROR engine | worker crash: text/layers/3 prefill columns=1920: std::bad_alloc
FATAL engine unavailable after an engine-wide failure
The attention workspace is planned against the configured capacity but launched with the keys the
call actually sees. The Volta flash prefill route keeps a staging budget (131,072 keys), and the
planner asked whether the capacity was inside it: with --max-context 200000 it concluded no and
reserved nothing, while every call shorter than the budget still selected the route and asked the
arena for up to 375 MiB it had never planned — so the first attention layer to run one threw. The
reservation now covers the largest key count at which the route still runs, the same way the fast
NVFP4 prompt branch reserves at the last key count its route holds for. Configurations at or below
the budget were never affected, which is why every measurement above (all at --max-context 131072) passed.
tests/ops/test_causal_attention_workspace.cpp pins the invariant: for capacities of 131,072,
200,000 and 262,144 keys, the planned reservation must cover a call at every visible key count
below it, for both the BF16 and INT8 caches. Against the previous planner it reports the failure
directly (capacity 200000 plans 0.0 MiB, a call at 83000 visible keys needs 374.5 MiB).
The reservation costs the arena what the route allocates anyway when it runs: 562 MiB at the
staging budget for this geometry. On a card where that competes with KV, Auto KV gives up roughly
17,000 int8 tokens out of 250,000.
What changed
- Copy drafting is opt-in.
--ngram-draft-tokensdefaults to 0 andspeculative.*counts only
neural rows; copy rounds are reported throughngram_*. The reported acceptance is now comparable
across configurations, which is what made the #2
numbers look like a 1.5 → 1.6 regression. This is the behaviour change in the release: a service
that relied on the old default loses the 15-token copy drafting it never named. - Upstream merge. 148 commits, including the verify pass that partitions KV by the live draft
window and the sm_70 FP16 saturation fix.NINFER_SM70becameNINFER_TIER_VOLTA(the
NINFER_SM70_*environment variables keep their names). - Volta decode routes. FP8 SwiGLU
gate_upis prepacked at load (294 → 111 µs a call), FP8
residual QPN activations stage through FP16 (62.9 → 47.4 µs), NVFP4 attention and GDN inputs join
the QPN path, and narrow Q8/Q4 calls use their own kernels (Q8 282 → 150 µs, Q4 1072 → 567 µs). - Long-context prefill. The vendored flash route kept a 32K visible-key staging cap, so every
chunk past 32K fell back to the sm_70 prompt kernel at 105 ms a call; at 66K that alone cost 32 s.
The cap is now 131,072 keys and the route runs 1024-token Q blocks. 66K prefill 710.8 → 838 tok/s,
27K 1000 → 1010. The staging planes come out of Auto KV: capacity at--max-context 131072drops
from 286,400 to 281,152 tokens, and free VRAM after startup is unchanged.
A failed allocation now also names the workspace it came from, so a report like the one above
carries (workspace used N MiB, peak N MiB of N MiB) beside the layer and column count.
Known costs
- Prefill at 66K is 1.0% behind 1.5. The attribution is measured, not guessed: the exact NVFP4
decode costs 645 ms more than 1.5's two-rounding FP16 shift trick over an 80 s prefill (that trick
fails this line's A16 criterion, relative L2 3.3e-3), and the prompt is split into 1920-token
attention steps plus a 128-token tail, which issues 8% more flash launches than 1.5's 2048-token
steps. Per request the decode gain dominates: a 27K prompt with 500 output tokens finishes 2.5%
sooner, a 66K prompt 0.3% sooner. --spec mtpoutput is not token-identical to running without speculation, on this line and on
1.5 alike: batched verification reaches near-tied logits differently. Use no--specwhen exact
equivalence to an unspeculated decode matters.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.6.2-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity |
ninfer-serve-sm70.tar.gz |
server binary only |
SHA256SUMS.txt |
checksums for both |
Documentation
The measured record, including the rejected approaches and the per-kernel attribution, is in
v1.6-vs-v1.5-2026-10-20.md.