Skip to content

Releases

NInfer V100 v1.7.2

Choose a tag to compare

@github-actions github-actions released this 04 Nov 14:36

NInfer V100 v1.7.2

Prebuilt Tesla V100 (sm_70) binary, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The V100 runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
The published image packages the sm_70 build. A local two-architecture image (sm_70 + sm_89,
Dockerfile.1.7.2.local) covers hosts that also run an RTX 40 series card.

Ampere: NVFP4 decode is memory-bound now (e2m1 shared-memory LUT)

Ada has no FP4 tensor cores, so the NVFP4 SIMT/GEMV kernels paid the __nv_fp4x2_e2m1
intrinsic's 4–6-instruction ALU sequence for every 4-bit code. The decode is now a 16-entry
__shared__ LUT (bit-exact with the e2m1 spec) in all four NVFP4 decode kernels
(b1a1a1a). Swift1.5 27B NVFP4, RTX 4080 SUPER (sm_89),
--nvfp4-native:

mode 1.7.1 1.7.2 change
plain decode (tg512) 19.97 34.86 +75%
DFlash2 K5 (tg512) 41.89 66.92 +60%

Post-change profiling shows the NVFP4 kernels at 68–70% of DRAM throughput — the arithmetic
is no longer the limit. Quality is unchanged (bit-exact; perplexity identical).

Ampere: load-time NVFP4 → row-split Q4 transcode, and --nvfp4-native

Ampere-tier loads now transcode NVFP4 tensors to Q4_G64_FP16 row-split at load time
(95e1f37), which suits mixed-format W4A4 artifacts
(+12.3% decode on the Coder390 W4A4). Pure-NVFP4 artifacts (Swift1.5) are faster native,
so --nvfp4-native (CLI, serve, bench, perplexity) keeps the stored NVFP4 path — and with
the LUT above, native is the right choice there (+119% over transcode). The flag is ignored
on Volta, which always runs its native QPN route.

Flash-Next (V100): CPU experts skip the bus, and the n-gram table is safe

  • Hybrid execution: CPU threads now compute their expert share straight from the host copy
    instead of staging it over PCIe first (ee2ef85), with the
    expert-miss staging path attributed in the bench. Measured final configuration and lever
    space are in docs/performance.md.
  • The n-gram table no longer garbles output: untrained/uninitialised rows are zeroed and
    non-finite rows rejected (0aa1761).
  • Converter: the qwen4_exp GGUF recipe accepts ggml block-stored GDN controls and older MTP
    block encodings (64f6c5c, 229fe30).
  • 200,000-token context with --kv-dtype rk4v4 without losing expert cache
    (+0.040% perplexity at that setting).
  • Optional ggml-cpu fast path for host experts (20aa77a), off by default.

Build and diagnostics

  • Multi-architecture builds link again, and Volta-only units compile for sm_70 alone
    (5062dcc) — a fat 70;89 build was broken before.
  • NINFER_ROUND_TIMING=1 logs per-decode-round device/finish/commit wall time
    (ab24fd1); default off, zero overhead.

Documentation

The README is now the V100-32GB single-card document, and docs/performance.md carries only
this line's own measurements (V100-32GB and RTX 4080 SUPER), including the 512-token decode
protocol — the 128-token figures understated the rate because the draft head's acceptance
rises over the sequence.

Install

Docker (V100):

docker pull ghcr.io/jimmymax/ninfer-v100:1.7.2

Or unpack ninfer-v100-v1.7.2-linux-x86_64-sm70.tar.gz and run bin/ninfer-serve on a host
with CUDA 12.8+ driver libraries. See README.md for the measured artifact table and the
serving commands.

NInfer V100 v1.7.1

Choose a tag to compare

@JimmyMax JimmyMax released this 27 Oct 05:05

NInfer V100 v1.7.1

Prebuilt Tesla V100 (sm_70) binary, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The V100 runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
The same image also carries an sm_89 binary (RTX 40 series); a dispatch script in
/usr/local/bin/ninfer-serve selects the right one from the visible GPU's compute
capability, so one image serves both cards. NINFER_IMAGE_ARCH=sm70|sm89 overrides the
choice.

NVFP4 MTP acceptance restored

The Q6_K dense-prepack change in this release pulled the proposal head into the Volta
device-layout list unconditionally. On NVFP4 artifacts the proposal head is Q4_G64, so it
got panel-prepacked, the draft scores broke, MTP acceptance fell to ~0 (decode dropped to
~28 tok/s) and DFlash2 failed at startup. The layout pass now takes the proposal head only
when it is actually Q6_K (0311d39).

NVFP4 27B, V100, int8 KV, --spec mtp --draft-tokens 3: 65.5 tok/s (44.2% acceptance)
after the fix, vs ~28 tok/s before. --full-draft-head is no longer needed as a workaround.

GGUF Q6_K projections prepack into a dense 6-bit stream

Of a Q4_K_M 27B artifact's weights, Q6_K is ~26% and its vec kernel reads 210-byte rows
that are only 2-byte aligned, which caps the load width at 54–63% of the card's bandwidth.
The projections (dense and MTP, ~4.35 GiB of a 16.95 GiB artifact) are now repacked at load
time, device-side and in place, into a same-size dense 6-bit stream the vec kernel reads
with 64-bit loads (bd052b2). MoE experts keep the standard
layout; the permute is a no-op for artifacts without Q6_K projections, so NVFP4 artifacts
are unaffected. Artifacts whose recipes store a fused [gate; up] Q6_K parent (for example
the IQ3_S 27B, whose MTP layer stores gate and up row-contiguously in one object) keep that
parent in the stored layout: the fused swiglu kernel family reads standard GGUF blocks only,
and the dense permute is per-object (fused parents stay standard fix below).

A follow-up fix keeps exactly those fused Q6_K parents out of the prepack. Before it, such an
artifact failed to start on sm_70 with gguf swiglu: dense Q6_K parents must be non-fused,
because the prepack selected the shared parent object and permuted it while the fused swiglu
route could not read the dense layout.

Q4_K_M 27B, V100, int8 KV:

mode 1.7.0 1.7.1 change
plain decode (tg128) 35.1 38.1 +8.7%
MTP d3 decode (tg128) 54.1 57.2 +5.7%

vs v1.5.0 (NVFP4 27B artifact, greedy, INT8 group-64 KV, --max-context 200000, one V100-32GB)

Matched to the v1.5.0 release protocol: --spec mtp --draft-tokens 4, 1024 generated
tokens per run, four prompts of the same content on both builds:

prompt tokens v1.5.0 decode v1.5.0 round v1.7.1 decode v1.7.1 round
code 7,093 81.8 43.7 ms 81.7 40.4 ms
code 30,837 75.3 46.6 ms 82.3 42.8 ms
Chinese 6,365 77.7 43.9 ms 86.4 40.5 ms
Chinese 25,485 73.7 45.6 ms 81.0 42.4 ms

The round time is the drift-free quantity: v1.7.1 is 7–8% faster per round at every
length, so at equal draft acceptance it decodes faster. On three of the four prompts it
also takes the acceptance (MTP d4: 57.8–62.2% vs 58.8–63.3%, prompt-dependent trajectory
drift, both directions) and wins decode by 9–11%; on the 7K code prompt the acceptance
runs 6.5 pp lower and exactly offsets the round gain (81.7 vs 81.8).

The v1.5.0 release's headline numbers (101.1 @ 7.4K code, 94.8 @ 31.5K code) are
acceptance-driven and prompt-specific: this same v1.5.0 build gives 81.8 and 75.3 on
the code prompts above, so the gap to those figures is prompt predictability, not a
version regression. Prefill: 1.07k tok/s at 7K on both builds; at 27K, 957.7 (v1.7.1)
vs 965.5 (v1.5.0) at chunk 2048 and 982.0 (v1.7.1) vs 965.6 (v1.5.0) at chunk 4096.

Build

Dual-arch (70;89) compilation of this tree is pathologically slow on a few int8/splitk
units (30–174× on the sm_89 ptxas pass, ~10 h total), so the image is built from two
single-arch trees and packaged with the dispatcher above. Runtime behavior is identical to
a dual-arch build. The Volta kernel guards gained an explicit __CUDA_ARCH__ == 700
clause so the tree also compiles cleanly as one unit for several architectures
(ec1d53a).

Upgrade

  • V100 services: image: ninfer:1.7.1, no other changes.
  • NVFP4 artifact MTP: drop --full-draft-head if you added it as a workaround.
  • Rollback: ninfer:1.7.0.

NInfer V100 v1.7.0

Choose a tag to compare

@github-actions github-actions released this 27 Oct 06:32

NInfer V100 v1.7.0

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

The release line for faster GGML artifacts: every GGUF-family quantization this engine runs
(Q4_K/Q5_K/Q6_K/Q8_0 superblocks, IQ2/IQ3, Q4_0/Q5_0, the dense MoE mix of a standard
llama.cpp export) now decodes near its memory-bandwidth limit, and the converter accepts a
standard llama.cpp Q4_K_M GGUF of the Qwen3.8-27B family out of the box.

Fastest measured configuration (this release): the NVFP4 artifact with
--prefill-chunk 2048 --spec mtp --draft-tokens 4 --lm-head-draft reaches 84.7 tok/s
(58.9% acceptance) at a ~1.0K-token prompt and 75.5 tok/s (52.0% acceptance) at a 27.3K
prompt — the highest throughput measured on this engine so far. The GGUF artifact's best is
--prefill-chunk 4096 with the same decode flags.

GGML decode is 18.9% faster than 1.6.2

The vector GEMV that runs every GGUF weight read carries the decode: one kernel per (type, T,
fused) instantiation, T = tokens per batch. On Volta the compiled slice loop for wide T
(T ≥ 2, i.e. every MTP verification and every batched call) spilled its instruction footprint
past the 16 KiB per-SM L1I, and the fetch stall ate most of the gain wide T should give. T = 1
was already fully unrolled; the fix is #pragma unroll 1 on the slice loop for T ≥ 2
(9f8edb7), leaving the T = 1 loop fully unrolled as before.

Measured on one V100-PCIE-32GB (250 W, ECC on), int8 KV, 27.3K-token prompt, 96 output,
--spec mtp --draft-tokens 4 --lm-head-draft, GGUF Q4_K_M artifact:

mode 1.6.2 (pre-fix) 1.7.0 change
plain decode 29.3 32.0 +9.2%
MTP d4 decode 57.0 67.8 +18.9%

The op-level GEMV bandwidth per type after the fix (isolated, cold L2): Q4_K 644 GB/s (80% of
the card's 803 GB/s), Q8_0 600–700 GB/s (75–87%), Q6_K 430–506 GB/s (54–63% — its 210-byte rows
are only 2-byte aligned, which caps how wide the loads can be), IQ3_S 527–542 GB/s.

NVFP4 vs GGML on this release, and the 1.5.0 baseline

Same Swift-1.5 v3 weights in two representations, greedy, 96 output, int8 KV, one V100-32GB.
Short prompt = 982 tokens; long prompt = 27.2K tokens (the 27.3K rows come from the 40-case
tuning matrix at 27,250 tokens). Each side at its measured-best --prefill-chunk
(2048 NVFP4 / 4096 GGUF).

prompt mode 1.7.0 NVFP4 1.7.0 GGUF Q4_K_M 1.5.0 NVFP4 1.5.0 GGUF
short plain decode 34.3 33.3 33.9 n/a
short MTP d4 decode 84.7 (58.9%) 68.4 (51.6%) 77.6 (52.5%) n/a
short prefill 1,000 786 960 n/a
27.2K MTP d4 decode 68.3 (50.0%) 59.5 (45.5%) 75.8 (54.6%) n/a
27.2K prefill 975 969 977 n/a
27.3K plain decode 32.7 31.8 — n/a
27.3K MTP d4 decode 75.5 (52.0%) 73.8 (58.9%) — n/a
82K MTP d4 decode 67.0 (52.0%) 63.8 (55.1%) — n/a
82K prefill 754 753 (ch 4096) — n/a

Reading this table:

  • NVFP4 vs GGUF on 1.7.0: plain decode tied within 3%; MTP decode within 2.3–12% depending
    on prompt (the acceptance-rate spread dominates — see below); 82K prefill tied with chunk
    4096. The GGUF artifact is 2.2 GiB smaller on the GPU.
  • 1.7.0 vs 1.5.0 (same NVFP4 artifact): at the short prompt 1.7.0 is faster (84.7 vs 77.6,
    +9.1%); at the 27.2K prompt 1.5.0 is faster (75.8 vs 68.3, +11.0%). Both gaps are driven
    mostly by the acceptance rate, which is a property of the engine's draft/target numerics —
    the same artifact and prompt verify differently on the two engines (58.9/50.0% on 1.7.0
    vs 52.5/54.6% on 1.5.0). Per-round time is within ~5% between the two.
  • 1.5.0 GGUF is n/a: the 1.5.0 loader predates the v3 GGUF framing's F32 GDN controls and
    rejects the artifact at startup (ERROR startup failed | inspecting artifact). 1.5.0 never
    ran this GGUF; the comparison is against its NVFP4 route.

The remaining GGUF gap to NVFP4 is the representation, not the kernel: Q4_K decodes at 80% of
peak already, and the measured remainder decomposes into multi-stream contention on the Q4_K
kernels (~6 ms/token), the Q6_K narrow-load limitation above (~2–3 ms/token), and
attention/KV bandwidth (~5 ms/token at 27K). A conversion-side repack of the Q6_K rows could
recover ~2–3% more; nothing cheaper is left. See the tuning-matrix record for the full
breakdown.

What changed since 1.6.2

  • GGML vec decode: #pragma unroll 1 on the T ≥ 2 slice loop (the 18.9% above); T = 1
    unchanged.

  • Converter: standard llama.cpp Q4_K_M GGUFs of this model family convert with their F32
    GDN controls, so a published GGUF no longer needs a re-quantization to reach this engine.
    Canonical command:

    python3 -m tools.convert <model>.gguf --out <model>.ninfer --mmproj <model>-mmproj-F16.gguf
  • Everything from 1.6.2: the long---max-context std::bad_alloc fix, the flash prefill cap at
    200,000, opt-in --ngram-draft-tokens (default 0), and the Volta decode route set.

Fastest measured configuration (V100-32GB, this release)

ninfer-serve model.ninfer --host 0.0.0.0 --port 8000 --max-context 200000 \
  --prefill-chunk 2048 --kv-capacity auto --kv-dtype int8 --max-concurrency 1 \
  --spec mtp --draft-tokens 4 --lm-head-draft

NVFP4 artifact: 84.7 tok/s MTP decode at a ~1.0K prompt (58.9% acceptance), 75.5 tok/s at a
27.3K prompt (52.0%), 32.7 tok/s plain at 27.3K, 1,006 tok/s prefill at 27.3K, 67.0 tok/s at
82K. GGUF artifact: same flags with --prefill-chunk 4096 (988 tok/s prefill at 27.3K); 73.8
tok/s MTP at 27.3K, 63.8 tok/s at 82K.

Known costs

  • MTP decode remains 2.3–12% behind NVFP4 on the same weights depending on prompt; the gap is
    the quantization representation plus the artifact's frozen-draft-head acceptance spread
    (±5 pp across prompts and chunking), not a defect of the GGML path.
  • --spec mtp output is not token-identical to running without speculation, on this line and
    on 1.5 alike: batched verification reaches near-tied logits differently. Use no --spec
    when exact equivalence to an unspeculated decode matters.
  • 1.5.0 cannot load v3 GGUF artifacts (F32 GDN controls); upgrading to 1.7.0 is required for
    the GGML route on those artifacts.

Assets

Built locally; not pushed.

Asset Value
image ninfer:1.7.0 sha256:dd3a7dbb11e6
ninfer-serve binary sha256:2fb0c0a3f654ec0749d6f982177fa5f5a116b29ef94f332ee0d70b0687919475

Documentation

NInfer V100 v1.6.2

Choose a tag to compare

@github-actions github-actions released this 27 Oct 06:30

NInfer V100 v1.6.2

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

A performance release for the report in #2: the
1.6 line is now faster than 1.5 everywhere the reporter measured the loss. Nine commits merge
upstream's 148 post-1.6.1 commits and add the V100 route work; measured on one V100-PCIE-32GB
(250 W, ECC on), int8 KV, --prefill-chunk 2048, alternating A/B against 1.6.1.

Speculative decode is faster than 1.5 at every depth

--spec mtp --draft-tokens 4 --lm-head-draft, Swift v2 NVFP4 artifact:

prompt 1.5 1.6.2 change acceptance 1.5 → 1.6.2
502 65.7 70.7 +7.6% 40.5% → 43.2%
2,942 62.8 65.4 +4.1% 39.2% → 40.1%
27,222 53.3 59.2 +11.1% 32.7% → 36.9%

A second, native-nvfp4 artifact (orcarouter-Qwen3.8-27B-Uncensored-nvfp4) gains +2.8%, +2.4% and
+4.7% at the same depths, so the win is not specific to a GGUF-derived artifact. Plain decode is at
parity, and prefill is at parity at 27K and 1.0% behind at 66K.

Fixed: std::bad_alloc in prefill with a long --max-context

A configuration with --max-context past 131,072 crashed on its first long prefill:

ERROR engine | worker crash: text/layers/3 prefill columns=1920: std::bad_alloc
FATAL engine unavailable after an engine-wide failure

The attention workspace is planned against the configured capacity but launched with the keys the
call actually sees. The Volta flash prefill route keeps a staging budget (131,072 keys), and the
planner asked whether the capacity was inside it: with --max-context 200000 it concluded no and
reserved nothing, while every call shorter than the budget still selected the route and asked the
arena for up to 375 MiB it had never planned — so the first attention layer to run one threw. The
reservation now covers the largest key count at which the route still runs, the same way the fast
NVFP4 prompt branch reserves at the last key count its route holds for. Configurations at or below
the budget were never affected, which is why every measurement above (all at --max-context 131072) passed.

tests/ops/test_causal_attention_workspace.cpp pins the invariant: for capacities of 131,072,
200,000 and 262,144 keys, the planned reservation must cover a call at every visible key count
below it, for both the BF16 and INT8 caches. Against the previous planner it reports the failure
directly (capacity 200000 plans 0.0 MiB, a call at 83000 visible keys needs 374.5 MiB).

The reservation costs the arena what the route allocates anyway when it runs: 562 MiB at the
staging budget for this geometry. On a card where that competes with KV, Auto KV gives up roughly
17,000 int8 tokens out of 250,000.

What changed

  • Copy drafting is opt-in. --ngram-draft-tokens defaults to 0 and speculative.* counts only
    neural rows; copy rounds are reported through ngram_*. The reported acceptance is now comparable
    across configurations, which is what made the #2
    numbers look like a 1.5 → 1.6 regression. This is the behaviour change in the release: a service
    that relied on the old default loses the 15-token copy drafting it never named.
  • Upstream merge. 148 commits, including the verify pass that partitions KV by the live draft
    window and the sm_70 FP16 saturation fix. NINFER_SM70 became NINFER_TIER_VOLTA (the
    NINFER_SM70_* environment variables keep their names).
  • Volta decode routes. FP8 SwiGLU gate_up is prepacked at load (294 → 111 µs a call), FP8
    residual QPN activations stage through FP16 (62.9 → 47.4 µs), NVFP4 attention and GDN inputs join
    the QPN path, and narrow Q8/Q4 calls use their own kernels (Q8 282 → 150 µs, Q4 1072 → 567 µs).
  • Long-context prefill. The vendored flash route kept a 32K visible-key staging cap, so every
    chunk past 32K fell back to the sm_70 prompt kernel at 105 ms a call; at 66K that alone cost 32 s.
    The cap is now 131,072 keys and the route runs 1024-token Q blocks. 66K prefill 710.8 → 838 tok/s,
    27K 1000 → 1010. The staging planes come out of Auto KV: capacity at --max-context 131072 drops
    from 286,400 to 281,152 tokens, and free VRAM after startup is unchanged.

A failed allocation now also names the workspace it came from, so a report like the one above
carries (workspace used N MiB, peak N MiB of N MiB) beside the layer and column count.

Known costs

  • Prefill at 66K is 1.0% behind 1.5. The attribution is measured, not guessed: the exact NVFP4
    decode costs 645 ms more than 1.5's two-rounding FP16 shift trick over an 80 s prefill (that trick
    fails this line's A16 criterion, relative L2 3.3e-3), and the prompt is split into 1920-token
    attention steps plus a 128-token tail, which issues 8% more flash launches than 1.5's 2048-token
    steps. Per request the decode gain dominates: a 27K prompt with 500 output tokens finishes 2.5%
    sooner, a 66K prompt 0.3% sooner.
  • --spec mtp output is not token-identical to running without speculation, on this line and on
    1.5 alike: batched verification reaches near-tied logits differently. Use no --spec when exact
    equivalence to an unspeculated decode matters.

Assets

File Contents
ninfer-v100-v1.6.2-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity
ninfer-serve-sm70.tar.gz server binary only
SHA256SUMS.txt checksums for both

Documentation

The measured record, including the rejected approaches and the per-kernel attribution, is in
v1.6-vs-v1.5-2026-10-20.md.

NInfer V100 v1.6.1

Choose a tag to compare

@github-actions github-actions released this 19 Oct 08:09

NInfer V100 v1.6.1

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

A patch release on v1.6.0: one build
fix, the dense Gemma 4 31B, and the container image the instructions name.

Fixed: building from source other than Release

Building the tree in any configuration that keeps the vendored ggml CUDA host helpers —
CMAKE_BUILD_TYPE=Debug, -O0, -lineinfo — failed at the link step with

undefined reference to `ggml_cuda_get_device()'
undefined reference to `ggml_cuda_error(...)'

third_party/ggml-quants ships the ggml CUDA headers but not the common.cu that defines those two
functions, and only ggml_abort had a local definition. Release builds linked anyway because -O3
drops the unused inline callers that would reference them, so CI and the published binaries never
saw it — report: #2. Both are now defined beside
ggml_abort in src/ops/linear/gguf/ggml_bridge.cu, and the new
tests/ops/test_vendor_host_helpers.cpp links them from a static initializer, so deleting either
definition fails the test build at every optimisation level instead of failing a user's.

The v1.6.0 release binaries are not affected. They contain neither the call nor the reference;
only a build that keeps the unused callers needed the definitions.

Added: the dense Gemma 4 31B

The Gemma 4 row kernels capped hidden at 4096 (1024 float4 lanes per block), which the 31B's 5376
does not fit, so a dense 31B artifact stopped with "gemma4 row kernels: hidden must be whole
float4s, at most 4096"
. Wide-hidden variants of rms_norm, attn_residual and ffn_combine now
cover it, dispatched by hidden/4 > 1024 — the existing narrow kernels are untouched, so every
artifact that ran before runs exactly as before. The new FP64 oracle
(tests/models/gemma4/test_wide_rows.cpp) checks both widths against the same reference, and the
26B forced through the wide path reproduces its narrow-path output token for token.

Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP (Q4_K_M blocks, 17.6 GiB, dense, 60 layers,
hidden 5376), one V100-32GB, 4K context, greedy:

prefill decode MTP acceptance
--spec mtp --draft-tokens 4 674 tok/s 48.8 tok/s 43.0%
--spec mtp --draft-tokens 3 674 tok/s 49.7 tok/s 56.4%
no speculation 674 tok/s 27.8 tok/s —

Convert it with python3 -m tools.convert --recipe gemma4_gguf --components text,mtp,vision; draft
three pays on this model because a wider verification batch costs more weight traffic per accepted
token than it saves.

Changed: the V100 container image

The V100 build is ghcr.io/jimmymax/ninfer-v100:latest (:<version> pins a release, sm70 is an
alias), built from the new Dockerfile.sm70: CUDA 12.8 on Ubuntu 24.04, about 3 GB. The CUDA 13 image
for the RTX 30/40/50 line stays at :sm86-sm120a.

docker run --rm --gpus all -p 8000:8000 --ulimit memlock=-1 \
  -v "$PWD/models:/models" ghcr.io/jimmymax/ninfer-v100:latest \
  ninfer-serve /models/model.ninfer --host 0.0.0.0 --port 8000 --model-id model \
  --max-context 200000 --prefill-chunk 2048 --kv-dtype int8 \
  --spec mtp --draft-tokens 4 --lm-head-draft --ngram-draft-tokens 0

sm86-sm120a stays the sm_86/sm_120a image, which no V100 can run.

Fixed in the image, before anything else shipped

The first image this release published set ENTRYPOINT ["ninfer-serve"], so a compose command: list -- which names
the program itself, as every ninfer compose file does -- started ninfer-serve ninfer-serve /models/x.ninfer and the server stopped with

ninfer-serve: unknown argument: /workspace/models/x.ninfer

The image now sets no entrypoint, like the other ninfer images, and the commands above name
ninfer-serve first. 1.6.1, latest and sm70 are rebuilt with that fix: the binaries are the same,
and nothing else in this release changed.

Assets

File Contents
ninfer-v100-v1.6.1-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity
ninfer-serve-sm70.tar.gz server binary only
SHA256SUMS.txt checksums for both

Everything in the v1.6.0 highlights
holds: 27K-token prefill 1.00k tok/s, decode 78 tok/s, v2 artifacts read in place.

Documentation

docs/v100/gemma4-wide-rows-2026-10-16.md records the wide-row work, including the residual
added twice by writing the normalized activations back too early. docs/v100/PLAN.md is the
roadmap with one measurement document per stage.

Apache-2.0, as upstream; model weights keep their own licenses (see LICENSE and the README).

NInfer V100 v1.6.0

Choose a tag to compare

@JimmyMax JimmyMax released this 16 Oct 12:07

NInfer V100 v1.6.0

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

Read this first: this release is a different line from v1.0.0 – v1.5.0

v1.6.0 is the first release cut from main, which follows the NInfer-all upstream and adds the
single-card Tesla V100-32GB work. The earlier tags v1.0.0 … v1.5.0 were cut from the previous V100
line; that line now lives on as the archive/fork-master branch. It is not an ancestor of this
commit, so v1.6.0 is not a continuation of v1.5.0 and the two are not comparable line for line.

v1.0.0 … v1.5.0 (archive/fork-master) v1.6.0 (main)
Base previous V100 line NInfer-all upstream + the V100-32GB work below
Containers v2 (native vocabulary) v2 and v3
v2 artifacts read natively read in place, no offline conversion and no 24 GB scratch copy
GGUF-derived artifacts not supported — stops with unknown tensor format: gguf_iq4_xs supported through the built-in ggml bridge
Default branch, README, docs — main, with the V100-32GB line stated up front

Highlights

One Tesla V100-PCIE-32GB, a 27K-token prompt, int8 KV, MTP with four drafts, greedy, one request.

Artifact Container Weights Context 27K-token request
Qwen3.8-27B nvfp4 (native profile, production default) v3 19.1 GiB 254,208 TTFT 27.2 s, prefill 1.00k tok/s, decode 76–78 tok/s
The same family as a v2 file, read in place v2 22.1 GiB 200,000 TTFT 27.3 s, prefill 998 tok/s, decode 82.7 tok/s
Qwen3.8-27B IQ3_S (GGUF-derived, 18 block formats) v3 12.9 GiB 400,000 TTFT 30.8 s, prefill 885 tok/s, decode 66–74 tok/s

Short-context decode (1K prompt) is 82.3 tok/s, and MTP accepts 69% of its drafts on the 27K
prompt. Huihui-…-nvfp4 and orcarouter-…-nvfp4 (v2, the native family) and
Gemma4-26B-A4B-…-MTP (v3, GGUF-derived) load as well.

The container version does not affect speed — the same weights measure 998 tok/s as a v2 file and
997 tok/s as the upgraded v3 file. What the numbers track is the weight recipe: the native
NVFP4 / FP8 / group-wise formats get the prepacked and CUTLASS routes, while GGUF-derived artifacts
run through the ggml bridge with per-tensor block dequantization.

What changed

Six changes carry the numbers above. All of them are load-time or operator routing, none changes a
numeric contract, the v3 path is byte-identical to before, and each landed with the upstream FP64
oracles green (ctest) plus a 27K end-to-end A/B.

  • Volta QPN — quadpair-split-N prepacked GEMMs for nvfp4 and fp8, with wide T on CUTLASS over a
    dequantized staging plane: the projection routes go from 85 GB/s to 480–620 GB/s.
  • GDN prefill on the recurrent kernel — chunked and two-stage are a regression on sm_70: 2.9–3.3×
    for T = 1024…8192, and 3.75 s → 2.18 s out of the 27K prefill.
  • Flash-attention prefill kernel — paged-to-contiguous staging plus an sm_70 mma.sync kernel:
    41.1 ms → 25.1 ms for the same shape, and 27K TTFT 30.1 s → 27.2 s.
  • Speculative-decode configuration — --ngram-draft-tokens 0 and no --adaptive-mtp: the
    default fifteen n-gram drafts dilute the MTP acceptance rate. 27K decode 64.2 → 78.3 tok/s with
    byte-identical greedy output.
  • FP8 SwiGLU wide T on an FP32-output CUTLASS path — it used to fall into a four-token-chunk
    SIMT ladder: mixed-quantization prefill 508 → 998 tok/s.
  • v2 artifacts read in place — the directory is normalized to the v3 shape in memory and the
    weights are read where they are, so a v2 file and its upgraded copy give the same output and the
    same numbers.

Assets

File Contents
ninfer-v100-v1.6.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity
ninfer-serve-sm70.tar.gz server binary only
SHA256SUMS.txt checksums for both

Run the server on the V100 with:

./ninfer-serve /path/to/model.ninfer --host 0.0.0.0 --port 8000 --api-key sk-xxx \
  --max-context 200000 --prefill-chunk 2048 --kv-capacity auto --kv-dtype int8 \
  --spec mtp --draft-tokens 4 --lm-head-draft --ngram-draft-tokens 0

Documentation

docs/v100/PLAN.md is the roadmap with one measurement document per stage. The design records
behind this release: docs/v100/v2-direct-read-2026-10-16.md,
docs/v100/fp8-swiglu-wide-t-2026-10-16.md, docs/v100/volta-flash-prefill-2026-10-15.md,
docs/v100/gap-closure-2026-10-15.md and docs/v100/gdn-recurrent-and-prefill-gap-2026-10-15.md.

Apache-2.0, as upstream; model weights keep their own licenses (see LICENSE and the README).

NInfer V100 v1.5.0

Choose a tag to compare

@github-actions github-actions released this 11 Oct 12:50

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

Highlights

Decode is 13–28% faster than v1.3.0, from decode-side kernel work, a deeper speculative draft and a rewritten PV epilogue in the INT8 attention kernel.

Qwen3.8-27B nvfp4, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB, 1024 generated tokens per run.

Prompt tokens Prompt v1.3.0 (draft 3) v1.5.0 (draft 4) Change
7,353 code 81.1 101.1 +25%
7,792 Chinese 59.2 67.1 +13%
31,538 code 74.3 94.8 +28%
31,924 Chinese 54.8 57.5 +5%
186,420 code 51.3 62.2 +21%
193,104 Chinese 38.1 39.9 +5%

Numbers are decode tok/s. First token at 186K is unchanged from v1.3.0 (341 s, 547 tok/s cold prefill).

Round time is the drift-free quantity; the same runs give (code prompts):

round time 8K 32K 186K
v1.3.0, draft 3 39.5 ms 43.0 ms 63.0 ms
v1.4.0, draft 4 38.6 ms 41.3 ms 62.5 ms
v1.5.0, draft 4 38.3 ms 41.0 ms 60.0 ms

What changed since v1.3.0

  • Three decode-side ports from the same source: fp16 activation staging for the QPN residual and down projections (the kernel was converting every activation load), the NVFP4 residual add fused into the QPN epilogue, and a wider small-T split reduce that keeps several loads in flight per output element.
  • Chained fp16 PV accumulator in the INT8 small-T decode attention kernel: one accumulator per output chunk across the whole 64-key tile instead of a fresh accumulator and a float conversion per 8-key group. An ablation build shows that epilogue was 26% of the kernel while the tile loads were 8%, i.e. the kernel is instruction-issue bound rather than bandwidth bound. NINFER_SM70_ATTN_V3=0 restores the previous epilogue for in-image A/B; NINFER_SM70_ATTN_V2=0 still falls back to the original 4-warp kernel.
  • Draft depth: the deployment default moves to --draft-tokens 4, worth +9% to +17% over draft 3 at 8K–32K and +14% at 186K. Draft 5 is slower above 32K (118.8 ms per round at 186K).
  • Opt-in draft key window: NINFER_MTP_ATTN_WINDOW=<keys> lets the MTP layer attend to the last N keys only (+11% at 186K code, −7% at 32K, off by default). With speculation disabled it produces byte-identical output to window 0, so target-model attention is provably untouched.
  • Documentation fix, no behaviour change: the long-context prefill cost was misattributed to the KV materialization planner in an earlier draft of the plan document. Instrumented counters show the engine's per-chunk helpers are ~0 and the 149 s host part of the 337 s 186K prefill is eager per-layer submission inside the model's prefill execution. It is documented, not fixed, in this release.

Verification

  • ninfer_softmax_attention_test (public-contract oracle comparison) passes with NINFER_SM70_ATTN_V3 both on and off; ninfer_qwen3_6_runtime_mechanisms_test passes; the DFlash2 real-model integration test accepts 20/20 at K=7 B=2.
  • Every number above is paired with a repeated baseline in the same session; the drift control matched its first run exactly (8K 38.7/38.7 ms, 32K 41.5/41.5 ms, 186K 62.9/62.9 ms).

Assets

File Contents
ninfer-v100-v1.5.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license
ninfer-serve-sm70.tar.gz server binary only, ready for docker/README.md step 1
SHA256SUMS.txt checksums

Docker

docker pull ghcr.io/jimmymax/ninfer-v100:v1.5.0

See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table. For measurement protocol and the V100 performance plan see bench/v100/README.md and docs/V100-PERF-PLAN.md.

Full changelog: v1.3.0...v1.5.0

NInfer V100 v1.3.0

Choose a tag to compare

@github-actions github-actions released this 10 Oct 11:41

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

Highlights

Long-context decoding is 37–47% faster, and the first token arrives about two minutes sooner.

This release rewrites the INT8 small-T decode attention (key-split 8-warp Bc=64 with a byte_perm int8->fp16 dequant) and retunes the D256 flash prompt kernel.

Qwen3.8-27B nvfp4, MTP draft 3, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB. Decode is the mean of 2 seeds × 1024 generated tokens; first token is the cold prefill of the same prompt.

Prompt tokens Prompt Decode tok/s First token
7,353 code 78.2 → 81.1 (+3.7%) 6.8 s
7,792 Chinese 55.2 → 59.2 (+7.2%) 7.4 s
31,538 code 63.7 → 74.3 (+16.6%) 34.1 → 32.4 s
31,924 Chinese 50.3 → 54.8 (+9.0%) 34.5 → 32.9 s
186,420 code 37.4 → 51.3 (+37.3%) 461.7 → 341.1 s
193,104 Chinese 25.8 → 38.1 (+47.4%) 490.6 → 350.2 s

Gains scale with context because attention's share of decode grows with it. The decode gain is isolated to the new kernel by an in-image A/B switch — NINFER_SM70_ATTN_V2=0 restores the previous kernel — and the first-token gain comes from the prompt retune alone.

Other changes since v1.1.0

  • Fixes the DFlash prefill per-chunk binding and waits for device work before releasing resources (upstream ports; the binding bug could write into the wrong state slot).
  • Defaults to spin-based CUDA synchronization, configurable with NINFER_CUDA_SYNC.
  • Ports the upstream materialization search and adaptive planning-budget rewrite, including the budget oracle and resource-manager tests.
  • Adds a push-triggered CI workflow, the bench/v100/ long-context measurement clients and protocol, and docs/V100-PERF-PLAN.md for continued work.

Verification

  • ninfer_softmax_attention_test passes with the new kernel and with NINFER_SM70_ATTN_V2=0.
  • Runtime mechanism checks pass; the DFlash2 real-model test accepts 20/20.
  • Every benchmark above is paired with a re-run of the previous release on the same host (decode repeats within 1.1%).

Assets

File Contents
ninfer-v100-v1.3.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license
ninfer-serve-sm70.tar.gz server binary only, ready for docker/README.md step 1
SHA256SUMS.txt checksums

Docker

docker pull ghcr.io/jimmymax/ninfer-v100:v1.3.0

See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table.

Full changelog: v1.1.0...v1.3.0

NInfer V100 v1.1.0

Choose a tag to compare

@github-actions github-actions released this 08 Oct 12:44

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on
Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute
capability 7.0 — other GPUs are rejected at startup.

Assets

File Contents
ninfer-v100-v1.1.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license
ninfer-serve-sm70.tar.gz server binary only, ready for docker/README.md step 1

Docker

docker pull ghcr.io/JimmyMax/ninfer-v100:v1.1.0

See docker/README.md for the one-click compose
recipe and docs/V100-BUILD.md for the runtime
flags. Model .ninfer artifacts are downloaded separately from
Hugging Face / ModelScope — see the README model table.

Full Changelog: v1.0.1...v1.1.0