Repository navigation
Releases
NInfer V100 v1.7.2
NInfer V100 v1.7.2
Prebuilt Tesla V100 (sm_70) binary, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The V100 runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
The published image packages the sm_70 build. A local two-architecture image (sm_70 + sm_89,
Dockerfile.1.7.2.local) covers hosts that also run an RTX 40 series card.
Ampere: NVFP4 decode is memory-bound now (e2m1 shared-memory LUT)
Ada has no FP4 tensor cores, so the NVFP4 SIMT/GEMV kernels paid the __nv_fp4x2_e2m1
intrinsic's 4–6-instruction ALU sequence for every 4-bit code. The decode is now a 16-entry
__shared__ LUT (bit-exact with the e2m1 spec) in all four NVFP4 decode kernels
(b1a1a1a). Swift1.5 27B NVFP4, RTX 4080 SUPER (sm_89),
--nvfp4-native:
| mode | 1.7.1 | 1.7.2 | change |
|---|---|---|---|
| plain decode (tg512) | 19.97 | 34.86 | +75% |
| DFlash2 K5 (tg512) | 41.89 | 66.92 | +60% |
Post-change profiling shows the NVFP4 kernels at 68–70% of DRAM throughput — the arithmetic
is no longer the limit. Quality is unchanged (bit-exact; perplexity identical).
Ampere: load-time NVFP4 → row-split Q4 transcode, and --nvfp4-native
Ampere-tier loads now transcode NVFP4 tensors to Q4_G64_FP16 row-split at load time
(95e1f37), which suits mixed-format W4A4 artifacts
(+12.3% decode on the Coder390 W4A4). Pure-NVFP4 artifacts (Swift1.5) are faster native,
so --nvfp4-native (CLI, serve, bench, perplexity) keeps the stored NVFP4 path — and with
the LUT above, native is the right choice there (+119% over transcode). The flag is ignored
on Volta, which always runs its native QPN route.
Flash-Next (V100): CPU experts skip the bus, and the n-gram table is safe
- Hybrid execution: CPU threads now compute their expert share straight from the host copy
instead of staging it over PCIe first (ee2ef85), with the
expert-miss staging path attributed in the bench. Measured final configuration and lever
space are indocs/performance.md. - The n-gram table no longer garbles output: untrained/uninitialised rows are zeroed and
non-finite rows rejected (0aa1761). - Converter: the qwen4_exp GGUF recipe accepts ggml block-stored GDN controls and older MTP
block encodings (64f6c5c,229fe30). - 200,000-token context with
--kv-dtype rk4v4without losing expert cache
(+0.040% perplexity at that setting). - Optional ggml-cpu fast path for host experts (
20aa77a), off by default.
Build and diagnostics
- Multi-architecture builds link again, and Volta-only units compile for sm_70 alone
(5062dcc) — a fat70;89build was broken before. NINFER_ROUND_TIMING=1logs per-decode-round device/finish/commit wall time
(ab24fd1); default off, zero overhead.
Documentation
The README is now the V100-32GB single-card document, and docs/performance.md carries only
this line's own measurements (V100-32GB and RTX 4080 SUPER), including the 512-token decode
protocol — the 128-token figures understated the rate because the draft head's acceptance
rises over the sequence.
Install
Docker (V100):
docker pull ghcr.io/jimmymax/ninfer-v100:1.7.2
Or unpack ninfer-v100-v1.7.2-linux-x86_64-sm70.tar.gz and run bin/ninfer-serve on a host
with CUDA 12.8+ driver libraries. See README.md for the measured artifact table and the
serving commands.
NInfer V100 v1.7.1
NInfer V100 v1.7.1
Prebuilt Tesla V100 (sm_70) binary, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The V100 runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
The same image also carries an sm_89 binary (RTX 40 series); a dispatch script in
/usr/local/bin/ninfer-serve selects the right one from the visible GPU's compute
capability, so one image serves both cards. NINFER_IMAGE_ARCH=sm70|sm89 overrides the
choice.
NVFP4 MTP acceptance restored
The Q6_K dense-prepack change in this release pulled the proposal head into the Volta
device-layout list unconditionally. On NVFP4 artifacts the proposal head is Q4_G64, so it
got panel-prepacked, the draft scores broke, MTP acceptance fell to ~0 (decode dropped to
~28 tok/s) and DFlash2 failed at startup. The layout pass now takes the proposal head only
when it is actually Q6_K (0311d39).
NVFP4 27B, V100, int8 KV, --spec mtp --draft-tokens 3: 65.5 tok/s (44.2% acceptance)
after the fix, vs ~28 tok/s before. --full-draft-head is no longer needed as a workaround.
GGUF Q6_K projections prepack into a dense 6-bit stream
Of a Q4_K_M 27B artifact's weights, Q6_K is ~26% and its vec kernel reads 210-byte rows
that are only 2-byte aligned, which caps the load width at 54–63% of the card's bandwidth.
The projections (dense and MTP, ~4.35 GiB of a 16.95 GiB artifact) are now repacked at load
time, device-side and in place, into a same-size dense 6-bit stream the vec kernel reads
with 64-bit loads (bd052b2). MoE experts keep the standard
layout; the permute is a no-op for artifacts without Q6_K projections, so NVFP4 artifacts
are unaffected. Artifacts whose recipes store a fused [gate; up] Q6_K parent (for example
the IQ3_S 27B, whose MTP layer stores gate and up row-contiguously in one object) keep that
parent in the stored layout: the fused swiglu kernel family reads standard GGUF blocks only,
and the dense permute is per-object (fused parents stay standard fix below).
A follow-up fix keeps exactly those fused Q6_K parents out of the prepack. Before it, such an
artifact failed to start on sm_70 with gguf swiglu: dense Q6_K parents must be non-fused,
because the prepack selected the shared parent object and permuted it while the fused swiglu
route could not read the dense layout.
Q4_K_M 27B, V100, int8 KV:
| mode | 1.7.0 | 1.7.1 | change |
|---|---|---|---|
| plain decode (tg128) | 35.1 | 38.1 | +8.7% |
| MTP d3 decode (tg128) | 54.1 | 57.2 | +5.7% |
vs v1.5.0 (NVFP4 27B artifact, greedy, INT8 group-64 KV, --max-context 200000, one V100-32GB)
Matched to the v1.5.0 release protocol: --spec mtp --draft-tokens 4, 1024 generated
tokens per run, four prompts of the same content on both builds:
| prompt | tokens | v1.5.0 decode | v1.5.0 round | v1.7.1 decode | v1.7.1 round |
|---|---|---|---|---|---|
| code | 7,093 | 81.8 | 43.7 ms | 81.7 | 40.4 ms |
| code | 30,837 | 75.3 | 46.6 ms | 82.3 | 42.8 ms |
| Chinese | 6,365 | 77.7 | 43.9 ms | 86.4 | 40.5 ms |
| Chinese | 25,485 | 73.7 | 45.6 ms | 81.0 | 42.4 ms |
The round time is the drift-free quantity: v1.7.1 is 7–8% faster per round at every
length, so at equal draft acceptance it decodes faster. On three of the four prompts it
also takes the acceptance (MTP d4: 57.8–62.2% vs 58.8–63.3%, prompt-dependent trajectory
drift, both directions) and wins decode by 9–11%; on the 7K code prompt the acceptance
runs 6.5 pp lower and exactly offsets the round gain (81.7 vs 81.8).
The v1.5.0 release's headline numbers (101.1 @ 7.4K code, 94.8 @ 31.5K code) are
acceptance-driven and prompt-specific: this same v1.5.0 build gives 81.8 and 75.3 on
the code prompts above, so the gap to those figures is prompt predictability, not a
version regression. Prefill: 1.07k tok/s at 7K on both builds; at 27K, 957.7 (v1.7.1)
vs 965.5 (v1.5.0) at chunk 2048 and 982.0 (v1.7.1) vs 965.6 (v1.5.0) at chunk 4096.
Build
Dual-arch (70;89) compilation of this tree is pathologically slow on a few int8/splitk
units (30–174× on the sm_89 ptxas pass, ~10 h total), so the image is built from two
single-arch trees and packaged with the dispatcher above. Runtime behavior is identical to
a dual-arch build. The Volta kernel guards gained an explicit __CUDA_ARCH__ == 700
clause so the tree also compiles cleanly as one unit for several architectures
(ec1d53a).
Upgrade
- V100 services:
image: ninfer:1.7.1, no other changes. - NVFP4 artifact MTP: drop
--full-draft-headif you added it as a workaround. - Rollback:
ninfer:1.7.0.
NInfer V100 v1.7.0
NInfer V100 v1.7.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
The release line for faster GGML artifacts: every GGUF-family quantization this engine runs
(Q4_K/Q5_K/Q6_K/Q8_0 superblocks, IQ2/IQ3, Q4_0/Q5_0, the dense MoE mix of a standard
llama.cpp export) now decodes near its memory-bandwidth limit, and the converter accepts a
standard llama.cpp Q4_K_M GGUF of the Qwen3.8-27B family out of the box.
Fastest measured configuration (this release): the NVFP4 artifact with
--prefill-chunk 2048 --spec mtp --draft-tokens 4 --lm-head-draft reaches 84.7 tok/s
(58.9% acceptance) at a ~1.0K-token prompt and 75.5 tok/s (52.0% acceptance) at a 27.3K
prompt — the highest throughput measured on this engine so far. The GGUF artifact's best is
--prefill-chunk 4096 with the same decode flags.
GGML decode is 18.9% faster than 1.6.2
The vector GEMV that runs every GGUF weight read carries the decode: one kernel per (type, T,
fused) instantiation, T = tokens per batch. On Volta the compiled slice loop for wide T
(T ≥ 2, i.e. every MTP verification and every batched call) spilled its instruction footprint
past the 16 KiB per-SM L1I, and the fetch stall ate most of the gain wide T should give. T = 1
was already fully unrolled; the fix is #pragma unroll 1 on the slice loop for T ≥ 2
(9f8edb7), leaving the T = 1 loop fully unrolled as before.
Measured on one V100-PCIE-32GB (250 W, ECC on), int8 KV, 27.3K-token prompt, 96 output,
--spec mtp --draft-tokens 4 --lm-head-draft, GGUF Q4_K_M artifact:
| mode | 1.6.2 (pre-fix) | 1.7.0 | change |
|---|---|---|---|
| plain decode | 29.3 | 32.0 | +9.2% |
| MTP d4 decode | 57.0 | 67.8 | +18.9% |
The op-level GEMV bandwidth per type after the fix (isolated, cold L2): Q4_K 644 GB/s (80% of
the card's 803 GB/s), Q8_0 600–700 GB/s (75–87%), Q6_K 430–506 GB/s (54–63% — its 210-byte rows
are only 2-byte aligned, which caps how wide the loads can be), IQ3_S 527–542 GB/s.
NVFP4 vs GGML on this release, and the 1.5.0 baseline
Same Swift-1.5 v3 weights in two representations, greedy, 96 output, int8 KV, one V100-32GB.
Short prompt = 982 tokens; long prompt = 27.2K tokens (the 27.3K rows come from the 40-case
tuning matrix at 27,250 tokens). Each side at its measured-best --prefill-chunk
(2048 NVFP4 / 4096 GGUF).
| prompt | mode | 1.7.0 NVFP4 | 1.7.0 GGUF Q4_K_M | 1.5.0 NVFP4 | 1.5.0 GGUF |
|---|---|---|---|---|---|
| short | plain decode | 34.3 | 33.3 | 33.9 | n/a |
| short | MTP d4 decode | 84.7 (58.9%) | 68.4 (51.6%) | 77.6 (52.5%) | n/a |
| short | prefill | 1,000 | 786 | 960 | n/a |
| 27.2K | MTP d4 decode | 68.3 (50.0%) | 59.5 (45.5%) | 75.8 (54.6%) | n/a |
| 27.2K | prefill | 975 | 969 | 977 | n/a |
| 27.3K | plain decode | 32.7 | 31.8 | — | n/a |
| 27.3K | MTP d4 decode | 75.5 (52.0%) | 73.8 (58.9%) | — | n/a |
| 82K | MTP d4 decode | 67.0 (52.0%) | 63.8 (55.1%) | — | n/a |
| 82K | prefill | 754 | 753 (ch 4096) | — | n/a |
Reading this table:
- NVFP4 vs GGUF on 1.7.0: plain decode tied within 3%; MTP decode within 2.3–12% depending
on prompt (the acceptance-rate spread dominates — see below); 82K prefill tied with chunk
4096. The GGUF artifact is 2.2 GiB smaller on the GPU. - 1.7.0 vs 1.5.0 (same NVFP4 artifact): at the short prompt 1.7.0 is faster (84.7 vs 77.6,
+9.1%); at the 27.2K prompt 1.5.0 is faster (75.8 vs 68.3, +11.0%). Both gaps are driven
mostly by the acceptance rate, which is a property of the engine's draft/target numerics —
the same artifact and prompt verify differently on the two engines (58.9/50.0% on 1.7.0
vs 52.5/54.6% on 1.5.0). Per-round time is within ~5% between the two. - 1.5.0 GGUF is n/a: the 1.5.0 loader predates the v3 GGUF framing's F32 GDN controls and
rejects the artifact at startup (ERROR startup failed | inspecting artifact). 1.5.0 never
ran this GGUF; the comparison is against its NVFP4 route.
The remaining GGUF gap to NVFP4 is the representation, not the kernel: Q4_K decodes at 80% of
peak already, and the measured remainder decomposes into multi-stream contention on the Q4_K
kernels (~6 ms/token), the Q6_K narrow-load limitation above (~2–3 ms/token), and
attention/KV bandwidth (~5 ms/token at 27K). A conversion-side repack of the Q6_K rows could
recover ~2–3% more; nothing cheaper is left. See the tuning-matrix record for the full
breakdown.
What changed since 1.6.2
-
GGML vec decode:
#pragma unroll 1on the T ≥ 2 slice loop (the 18.9% above); T = 1
unchanged. -
Converter: standard llama.cpp
Q4_K_MGGUFs of this model family convert with their F32
GDN controls, so a published GGUF no longer needs a re-quantization to reach this engine.
Canonical command:python3 -m tools.convert <model>.gguf --out <model>.ninfer --mmproj <model>-mmproj-F16.gguf -
Everything from 1.6.2: the long-
--max-contextstd::bad_allocfix, the flash prefill cap at
200,000, opt-in--ngram-draft-tokens(default 0), and the Volta decode route set.
Fastest measured configuration (V100-32GB, this release)
ninfer-serve model.ninfer --host 0.0.0.0 --port 8000 --max-context 200000 \
--prefill-chunk 2048 --kv-capacity auto --kv-dtype int8 --max-concurrency 1 \
--spec mtp --draft-tokens 4 --lm-head-draftNVFP4 artifact: 84.7 tok/s MTP decode at a ~1.0K prompt (58.9% acceptance), 75.5 tok/s at a
27.3K prompt (52.0%), 32.7 tok/s plain at 27.3K, 1,006 tok/s prefill at 27.3K, 67.0 tok/s at
82K. GGUF artifact: same flags with --prefill-chunk 4096 (988 tok/s prefill at 27.3K); 73.8
tok/s MTP at 27.3K, 63.8 tok/s at 82K.
Known costs
- MTP decode remains 2.3–12% behind NVFP4 on the same weights depending on prompt; the gap is
the quantization representation plus the artifact's frozen-draft-head acceptance spread
(±5 pp across prompts and chunking), not a defect of the GGML path. --spec mtpoutput is not token-identical to running without speculation, on this line and
on 1.5 alike: batched verification reaches near-tied logits differently. Use no--spec
when exact equivalence to an unspeculated decode matters.- 1.5.0 cannot load v3 GGUF artifacts (F32 GDN controls); upgrading to 1.7.0 is required for
the GGML route on those artifacts.
Assets
Built locally; not pushed.
| Asset | Value |
|---|---|
image ninfer:1.7.0 |
sha256:dd3a7dbb11e6 |
ninfer-serve binary |
sha256:2fb0c0a3f654ec0749d6f982177fa5f5a116b29ef94f332ee0d70b0687919475 |
Documentation
- NVFP4 vs GGML A/B under 1.7.0, with the 1.5.0 baseline and the per-kernel / op-level
headroom analysis: ab-swiftv3-nvfp4-vs-gguf-2026-10-22.html - The vec optimization record (rejected variants included):
gguf-vec-optimization-2026-10-21.md - The tuning-matrix raw data, including this release's A/B runs:
bench/v100/data/swiftv3_tuning_2026-10-22/
NInfer V100 v1.6.2
NInfer V100 v1.6.2
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
A performance release for the report in #2: the
1.6 line is now faster than 1.5 everywhere the reporter measured the loss. Nine commits merge
upstream's 148 post-1.6.1 commits and add the V100 route work; measured on one V100-PCIE-32GB
(250 W, ECC on), int8 KV, --prefill-chunk 2048, alternating A/B against 1.6.1.
Speculative decode is faster than 1.5 at every depth
--spec mtp --draft-tokens 4 --lm-head-draft, Swift v2 NVFP4 artifact:
| prompt | 1.5 | 1.6.2 | change | acceptance 1.5 → 1.6.2 |
|---|---|---|---|---|
| 502 | 65.7 | 70.7 | +7.6% | 40.5% → 43.2% |
| 2,942 | 62.8 | 65.4 | +4.1% | 39.2% → 40.1% |
| 27,222 | 53.3 | 59.2 | +11.1% | 32.7% → 36.9% |
A second, native-nvfp4 artifact (orcarouter-Qwen3.8-27B-Uncensored-nvfp4) gains +2.8%, +2.4% and
+4.7% at the same depths, so the win is not specific to a GGUF-derived artifact. Plain decode is at
parity, and prefill is at parity at 27K and 1.0% behind at 66K.
Fixed: std::bad_alloc in prefill with a long --max-context
A configuration with --max-context past 131,072 crashed on its first long prefill:
ERROR engine | worker crash: text/layers/3 prefill columns=1920: std::bad_alloc
FATAL engine unavailable after an engine-wide failure
The attention workspace is planned against the configured capacity but launched with the keys the
call actually sees. The Volta flash prefill route keeps a staging budget (131,072 keys), and the
planner asked whether the capacity was inside it: with --max-context 200000 it concluded no and
reserved nothing, while every call shorter than the budget still selected the route and asked the
arena for up to 375 MiB it had never planned — so the first attention layer to run one threw. The
reservation now covers the largest key count at which the route still runs, the same way the fast
NVFP4 prompt branch reserves at the last key count its route holds for. Configurations at or below
the budget were never affected, which is why every measurement above (all at --max-context 131072) passed.
tests/ops/test_causal_attention_workspace.cpp pins the invariant: for capacities of 131,072,
200,000 and 262,144 keys, the planned reservation must cover a call at every visible key count
below it, for both the BF16 and INT8 caches. Against the previous planner it reports the failure
directly (capacity 200000 plans 0.0 MiB, a call at 83000 visible keys needs 374.5 MiB).
The reservation costs the arena what the route allocates anyway when it runs: 562 MiB at the
staging budget for this geometry. On a card where that competes with KV, Auto KV gives up roughly
17,000 int8 tokens out of 250,000.
What changed
- Copy drafting is opt-in.
--ngram-draft-tokensdefaults to 0 andspeculative.*counts only
neural rows; copy rounds are reported throughngram_*. The reported acceptance is now comparable
across configurations, which is what made the #2
numbers look like a 1.5 → 1.6 regression. This is the behaviour change in the release: a service
that relied on the old default loses the 15-token copy drafting it never named. - Upstream merge. 148 commits, including the verify pass that partitions KV by the live draft
window and the sm_70 FP16 saturation fix.NINFER_SM70becameNINFER_TIER_VOLTA(the
NINFER_SM70_*environment variables keep their names). - Volta decode routes. FP8 SwiGLU
gate_upis prepacked at load (294 → 111 µs a call), FP8
residual QPN activations stage through FP16 (62.9 → 47.4 µs), NVFP4 attention and GDN inputs join
the QPN path, and narrow Q8/Q4 calls use their own kernels (Q8 282 → 150 µs, Q4 1072 → 567 µs). - Long-context prefill. The vendored flash route kept a 32K visible-key staging cap, so every
chunk past 32K fell back to the sm_70 prompt kernel at 105 ms a call; at 66K that alone cost 32 s.
The cap is now 131,072 keys and the route runs 1024-token Q blocks. 66K prefill 710.8 → 838 tok/s,
27K 1000 → 1010. The staging planes come out of Auto KV: capacity at--max-context 131072drops
from 286,400 to 281,152 tokens, and free VRAM after startup is unchanged.
A failed allocation now also names the workspace it came from, so a report like the one above
carries (workspace used N MiB, peak N MiB of N MiB) beside the layer and column count.
Known costs
- Prefill at 66K is 1.0% behind 1.5. The attribution is measured, not guessed: the exact NVFP4
decode costs 645 ms more than 1.5's two-rounding FP16 shift trick over an 80 s prefill (that trick
fails this line's A16 criterion, relative L2 3.3e-3), and the prompt is split into 1920-token
attention steps plus a 128-token tail, which issues 8% more flash launches than 1.5's 2048-token
steps. Per request the decode gain dominates: a 27K prompt with 500 output tokens finishes 2.5%
sooner, a 66K prompt 0.3% sooner. --spec mtpoutput is not token-identical to running without speculation, on this line and on
1.5 alike: batched verification reaches near-tied logits differently. Use no--specwhen exact
equivalence to an unspeculated decode matters.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.6.2-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity |
ninfer-serve-sm70.tar.gz |
server binary only |
SHA256SUMS.txt |
checksums for both |
Documentation
The measured record, including the rejected approaches and the per-kernel attribution, is in
v1.6-vs-v1.5-2026-10-20.md.
NInfer V100 v1.6.1
NInfer V100 v1.6.1
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
A patch release on v1.6.0: one build
fix, the dense Gemma 4 31B, and the container image the instructions name.
Fixed: building from source other than Release
Building the tree in any configuration that keeps the vendored ggml CUDA host helpers —
CMAKE_BUILD_TYPE=Debug, -O0, -lineinfo — failed at the link step with
undefined reference to `ggml_cuda_get_device()'
undefined reference to `ggml_cuda_error(...)'
third_party/ggml-quants ships the ggml CUDA headers but not the common.cu that defines those two
functions, and only ggml_abort had a local definition. Release builds linked anyway because -O3
drops the unused inline callers that would reference them, so CI and the published binaries never
saw it — report: #2. Both are now defined beside
ggml_abort in src/ops/linear/gguf/ggml_bridge.cu, and the new
tests/ops/test_vendor_host_helpers.cpp links them from a static initializer, so deleting either
definition fails the test build at every optimisation level instead of failing a user's.
The v1.6.0 release binaries are not affected. They contain neither the call nor the reference;
only a build that keeps the unused callers needed the definitions.
Added: the dense Gemma 4 31B
The Gemma 4 row kernels capped hidden at 4096 (1024 float4 lanes per block), which the 31B's 5376
does not fit, so a dense 31B artifact stopped with "gemma4 row kernels: hidden must be whole
float4s, at most 4096". Wide-hidden variants of rms_norm, attn_residual and ffn_combine now
cover it, dispatched by hidden/4 > 1024 — the existing narrow kernels are untouched, so every
artifact that ran before runs exactly as before. The new FP64 oracle
(tests/models/gemma4/test_wide_rows.cpp) checks both widths against the same reference, and the
26B forced through the wide path reproduces its narrow-path output token for token.
Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP (Q4_K_M blocks, 17.6 GiB, dense, 60 layers,
hidden 5376), one V100-32GB, 4K context, greedy:
| prefill | decode | MTP acceptance | |
|---|---|---|---|
--spec mtp --draft-tokens 4 |
674 tok/s | 48.8 tok/s | 43.0% |
--spec mtp --draft-tokens 3 |
674 tok/s | 49.7 tok/s | 56.4% |
| no speculation | 674 tok/s | 27.8 tok/s | — |
Convert it with python3 -m tools.convert --recipe gemma4_gguf --components text,mtp,vision; draft
three pays on this model because a wider verification batch costs more weight traffic per accepted
token than it saves.
Changed: the V100 container image
The V100 build is ghcr.io/jimmymax/ninfer-v100:latest (:<version> pins a release, sm70 is an
alias), built from the new Dockerfile.sm70: CUDA 12.8 on Ubuntu 24.04, about 3 GB. The CUDA 13 image
for the RTX 30/40/50 line stays at :sm86-sm120a.
docker run --rm --gpus all -p 8000:8000 --ulimit memlock=-1 \
-v "$PWD/models:/models" ghcr.io/jimmymax/ninfer-v100:latest \
ninfer-serve /models/model.ninfer --host 0.0.0.0 --port 8000 --model-id model \
--max-context 200000 --prefill-chunk 2048 --kv-dtype int8 \
--spec mtp --draft-tokens 4 --lm-head-draft --ngram-draft-tokens 0sm86-sm120a stays the sm_86/sm_120a image, which no V100 can run.
Fixed in the image, before anything else shipped
The first image this release published set ENTRYPOINT ["ninfer-serve"], so a compose command: list -- which names
the program itself, as every ninfer compose file does -- started ninfer-serve ninfer-serve /models/x.ninfer and the server stopped with
ninfer-serve: unknown argument: /workspace/models/x.ninfer
The image now sets no entrypoint, like the other ninfer images, and the commands above name
ninfer-serve first. 1.6.1, latest and sm70 are rebuilt with that fix: the binaries are the same,
and nothing else in this release changed.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.6.1-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity |
ninfer-serve-sm70.tar.gz |
server binary only |
SHA256SUMS.txt |
checksums for both |
Everything in the v1.6.0 highlights
holds: 27K-token prefill 1.00k tok/s, decode 78 tok/s, v2 artifacts read in place.
Documentation
docs/v100/gemma4-wide-rows-2026-10-16.md records the wide-row work, including the residual
added twice by writing the normalized activations back too early. docs/v100/PLAN.md is the
roadmap with one measurement document per stage.
Apache-2.0, as upstream; model weights keep their own licenses (see LICENSE and the README).
NInfer V100 v1.6.0
NInfer V100 v1.6.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
Read this first: this release is a different line from v1.0.0 – v1.5.0
v1.6.0 is the first release cut from main, which follows the NInfer-all upstream and adds the
single-card Tesla V100-32GB work. The earlier tags v1.0.0 … v1.5.0 were cut from the previous V100
line; that line now lives on as the archive/fork-master branch. It is not an ancestor of this
commit, so v1.6.0 is not a continuation of v1.5.0 and the two are not comparable line for line.
v1.0.0 … v1.5.0 (archive/fork-master) |
v1.6.0 (main) |
|
|---|---|---|
| Base | previous V100 line | NInfer-all upstream + the V100-32GB work below |
| Containers | v2 (native vocabulary) | v2 and v3 |
| v2 artifacts | read natively | read in place, no offline conversion and no 24 GB scratch copy |
| GGUF-derived artifacts | not supported — stops with unknown tensor format: gguf_iq4_xs |
supported through the built-in ggml bridge |
| Default branch, README, docs | — | main, with the V100-32GB line stated up front |
Highlights
One Tesla V100-PCIE-32GB, a 27K-token prompt, int8 KV, MTP with four drafts, greedy, one request.
| Artifact | Container | Weights | Context | 27K-token request |
|---|---|---|---|---|
Qwen3.8-27B nvfp4 (native profile, production default) |
v3 | 19.1 GiB | 254,208 | TTFT 27.2 s, prefill 1.00k tok/s, decode 76–78 tok/s |
| The same family as a v2 file, read in place | v2 | 22.1 GiB | 200,000 | TTFT 27.3 s, prefill 998 tok/s, decode 82.7 tok/s |
Qwen3.8-27B IQ3_S (GGUF-derived, 18 block formats) |
v3 | 12.9 GiB | 400,000 | TTFT 30.8 s, prefill 885 tok/s, decode 66–74 tok/s |
Short-context decode (1K prompt) is 82.3 tok/s, and MTP accepts 69% of its drafts on the 27K
prompt. Huihui-…-nvfp4 and orcarouter-…-nvfp4 (v2, the native family) and
Gemma4-26B-A4B-…-MTP (v3, GGUF-derived) load as well.
The container version does not affect speed — the same weights measure 998 tok/s as a v2 file and
997 tok/s as the upgraded v3 file. What the numbers track is the weight recipe: the native
NVFP4 / FP8 / group-wise formats get the prepacked and CUTLASS routes, while GGUF-derived artifacts
run through the ggml bridge with per-tensor block dequantization.
What changed
Six changes carry the numbers above. All of them are load-time or operator routing, none changes a
numeric contract, the v3 path is byte-identical to before, and each landed with the upstream FP64
oracles green (ctest) plus a 27K end-to-end A/B.
- Volta QPN — quadpair-split-N prepacked GEMMs for nvfp4 and fp8, with wide T on CUTLASS over a
dequantized staging plane: the projection routes go from 85 GB/s to 480–620 GB/s. - GDN prefill on the recurrent kernel — chunked and two-stage are a regression on sm_70: 2.9–3.3×
for T = 1024…8192, and 3.75 s → 2.18 s out of the 27K prefill. - Flash-attention prefill kernel — paged-to-contiguous staging plus an sm_70
mma.synckernel:
41.1 ms → 25.1 ms for the same shape, and 27K TTFT 30.1 s → 27.2 s. - Speculative-decode configuration —
--ngram-draft-tokens 0and no--adaptive-mtp: the
default fifteen n-gram drafts dilute the MTP acceptance rate. 27K decode 64.2 → 78.3 tok/s with
byte-identical greedy output. - FP8 SwiGLU wide T on an FP32-output CUTLASS path — it used to fall into a four-token-chunk
SIMT ladder: mixed-quantization prefill 508 → 998 tok/s. - v2 artifacts read in place — the directory is normalized to the v3 shape in memory and the
weights are read where they are, so a v2 file and its upgraded copy give the same output and the
same numbers.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.6.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity |
ninfer-serve-sm70.tar.gz |
server binary only |
SHA256SUMS.txt |
checksums for both |
Run the server on the V100 with:
./ninfer-serve /path/to/model.ninfer --host 0.0.0.0 --port 8000 --api-key sk-xxx \
--max-context 200000 --prefill-chunk 2048 --kv-capacity auto --kv-dtype int8 \
--spec mtp --draft-tokens 4 --lm-head-draft --ngram-draft-tokens 0Documentation
docs/v100/PLAN.md is the roadmap with one measurement document per stage. The design records
behind this release: docs/v100/v2-direct-read-2026-10-16.md,
docs/v100/fp8-swiglu-wide-t-2026-10-16.md, docs/v100/volta-flash-prefill-2026-10-15.md,
docs/v100/gap-closure-2026-10-15.md and docs/v100/gdn-recurrent-and-prefill-gap-2026-10-15.md.
Apache-2.0, as upstream; model weights keep their own licenses (see LICENSE and the README).
NInfer V100 v1.5.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
Highlights
Decode is 13–28% faster than v1.3.0, from decode-side kernel work, a deeper speculative draft and a rewritten PV epilogue in the INT8 attention kernel.
Qwen3.8-27B nvfp4, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB, 1024 generated tokens per run.
| Prompt tokens | Prompt | v1.3.0 (draft 3) | v1.5.0 (draft 4) | Change |
|---|---|---|---|---|
| 7,353 | code | 81.1 | 101.1 | +25% |
| 7,792 | Chinese | 59.2 | 67.1 | +13% |
| 31,538 | code | 74.3 | 94.8 | +28% |
| 31,924 | Chinese | 54.8 | 57.5 | +5% |
| 186,420 | code | 51.3 | 62.2 | +21% |
| 193,104 | Chinese | 38.1 | 39.9 | +5% |
Numbers are decode tok/s. First token at 186K is unchanged from v1.3.0 (341 s, 547 tok/s cold prefill).
Round time is the drift-free quantity; the same runs give (code prompts):
| round time | 8K | 32K | 186K |
|---|---|---|---|
| v1.3.0, draft 3 | 39.5 ms | 43.0 ms | 63.0 ms |
| v1.4.0, draft 4 | 38.6 ms | 41.3 ms | 62.5 ms |
| v1.5.0, draft 4 | 38.3 ms | 41.0 ms | 60.0 ms |
What changed since v1.3.0
- Three decode-side ports from the same source: fp16 activation staging for the QPN residual and down projections (the kernel was converting every activation load), the NVFP4 residual add fused into the QPN epilogue, and a wider small-T split reduce that keeps several loads in flight per output element.
- Chained fp16 PV accumulator in the INT8 small-T decode attention kernel: one accumulator per output chunk across the whole 64-key tile instead of a fresh accumulator and a float conversion per 8-key group. An ablation build shows that epilogue was 26% of the kernel while the tile loads were 8%, i.e. the kernel is instruction-issue bound rather than bandwidth bound.
NINFER_SM70_ATTN_V3=0restores the previous epilogue for in-image A/B;NINFER_SM70_ATTN_V2=0still falls back to the original 4-warp kernel. - Draft depth: the deployment default moves to
--draft-tokens 4, worth +9% to +17% over draft 3 at 8K–32K and +14% at 186K. Draft 5 is slower above 32K (118.8 ms per round at 186K). - Opt-in draft key window:
NINFER_MTP_ATTN_WINDOW=<keys>lets the MTP layer attend to the last N keys only (+11% at 186K code, −7% at 32K, off by default). With speculation disabled it produces byte-identical output to window 0, so target-model attention is provably untouched. - Documentation fix, no behaviour change: the long-context prefill cost was misattributed to the KV materialization planner in an earlier draft of the plan document. Instrumented counters show the engine's per-chunk helpers are ~0 and the 149 s host part of the 337 s 186K prefill is eager per-layer submission inside the model's prefill execution. It is documented, not fixed, in this release.
Verification
ninfer_softmax_attention_test(public-contract oracle comparison) passes withNINFER_SM70_ATTN_V3both on and off;ninfer_qwen3_6_runtime_mechanisms_testpasses; the DFlash2 real-model integration test accepts 20/20 atK=7 B=2.- Every number above is paired with a repeated baseline in the same session; the drift control matched its first run exactly (8K 38.7/38.7 ms, 32K 41.5/41.5 ms, 186K 62.9/62.9 ms).
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.5.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license |
ninfer-serve-sm70.tar.gz |
server binary only, ready for docker/README.md step 1 |
SHA256SUMS.txt |
checksums |
Docker
docker pull ghcr.io/jimmymax/ninfer-v100:v1.5.0See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table. For measurement protocol and the V100 performance plan see bench/v100/README.md and docs/V100-PERF-PLAN.md.
Full changelog: v1.3.0...v1.5.0
NInfer V100 v1.3.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
Highlights
Long-context decoding is 37–47% faster, and the first token arrives about two minutes sooner.
This release rewrites the INT8 small-T decode attention (key-split 8-warp Bc=64 with a byte_perm int8->fp16 dequant) and retunes the D256 flash prompt kernel.
Qwen3.8-27B nvfp4, MTP draft 3, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB. Decode is the mean of 2 seeds × 1024 generated tokens; first token is the cold prefill of the same prompt.
| Prompt tokens | Prompt | Decode tok/s | First token |
|---|---|---|---|
| 7,353 | code | 78.2 → 81.1 (+3.7%) | 6.8 s |
| 7,792 | Chinese | 55.2 → 59.2 (+7.2%) | 7.4 s |
| 31,538 | code | 63.7 → 74.3 (+16.6%) | 34.1 → 32.4 s |
| 31,924 | Chinese | 50.3 → 54.8 (+9.0%) | 34.5 → 32.9 s |
| 186,420 | code | 37.4 → 51.3 (+37.3%) | 461.7 → 341.1 s |
| 193,104 | Chinese | 25.8 → 38.1 (+47.4%) | 490.6 → 350.2 s |
Gains scale with context because attention's share of decode grows with it. The decode gain is isolated to the new kernel by an in-image A/B switch — NINFER_SM70_ATTN_V2=0 restores the previous kernel — and the first-token gain comes from the prompt retune alone.
Other changes since v1.1.0
- Fixes the DFlash prefill per-chunk binding and waits for device work before releasing resources (upstream ports; the binding bug could write into the wrong state slot).
- Defaults to spin-based CUDA synchronization, configurable with
NINFER_CUDA_SYNC. - Ports the upstream materialization search and adaptive planning-budget rewrite, including the budget oracle and resource-manager tests.
- Adds a push-triggered CI workflow, the
bench/v100/long-context measurement clients and protocol, anddocs/V100-PERF-PLAN.mdfor continued work.
Verification
ninfer_softmax_attention_testpasses with the new kernel and withNINFER_SM70_ATTN_V2=0.- Runtime mechanism checks pass; the DFlash2 real-model test accepts 20/20.
- Every benchmark above is paired with a re-run of the previous release on the same host (decode repeats within 1.1%).
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.3.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license |
ninfer-serve-sm70.tar.gz |
server binary only, ready for docker/README.md step 1 |
SHA256SUMS.txt |
checksums |
Docker
docker pull ghcr.io/jimmymax/ninfer-v100:v1.3.0See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table.
Full changelog: v1.1.0...v1.3.0
NInfer V100 v1.1.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on
Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute
capability 7.0 — other GPUs are rejected at startup.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.1.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license |
ninfer-serve-sm70.tar.gz |
server binary only, ready for docker/README.md step 1 |
Docker
docker pull ghcr.io/JimmyMax/ninfer-v100:v1.1.0See docker/README.md for the one-click compose
recipe and docs/V100-BUILD.md for the runtime
flags. Model .ninfer artifacts are downloaded separately from
Hugging Face / ModelScope — see the README model table.
Full Changelog: v1.0.1...v1.1.0