Repository navigation
NInfer V100 v1.6.1
NInfer V100 v1.6.1
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39).
The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
A patch release on v1.6.0: one build
fix, the dense Gemma 4 31B, and the container image the instructions name.
Fixed: building from source other than Release
Building the tree in any configuration that keeps the vendored ggml CUDA host helpers —
CMAKE_BUILD_TYPE=Debug, -O0, -lineinfo — failed at the link step with
undefined reference to `ggml_cuda_get_device()'
undefined reference to `ggml_cuda_error(...)'
third_party/ggml-quants ships the ggml CUDA headers but not the common.cu that defines those two
functions, and only ggml_abort had a local definition. Release builds linked anyway because -O3
drops the unused inline callers that would reference them, so CI and the published binaries never
saw it — report: #2. Both are now defined beside
ggml_abort in src/ops/linear/gguf/ggml_bridge.cu, and the new
tests/ops/test_vendor_host_helpers.cpp links them from a static initializer, so deleting either
definition fails the test build at every optimisation level instead of failing a user's.
The v1.6.0 release binaries are not affected. They contain neither the call nor the reference;
only a build that keeps the unused callers needed the definitions.
Added: the dense Gemma 4 31B
The Gemma 4 row kernels capped hidden at 4096 (1024 float4 lanes per block), which the 31B's 5376
does not fit, so a dense 31B artifact stopped with "gemma4 row kernels: hidden must be whole
float4s, at most 4096". Wide-hidden variants of rms_norm, attn_residual and ffn_combine now
cover it, dispatched by hidden/4 > 1024 — the existing narrow kernels are untouched, so every
artifact that ran before runs exactly as before. The new FP64 oracle
(tests/models/gemma4/test_wide_rows.cpp) checks both widths against the same reference, and the
26B forced through the wide path reproduces its narrow-path output token for token.
Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP (Q4_K_M blocks, 17.6 GiB, dense, 60 layers,
hidden 5376), one V100-32GB, 4K context, greedy:
| prefill | decode | MTP acceptance | |
|---|---|---|---|
--spec mtp --draft-tokens 4 |
674 tok/s | 48.8 tok/s | 43.0% |
--spec mtp --draft-tokens 3 |
674 tok/s | 49.7 tok/s | 56.4% |
| no speculation | 674 tok/s | 27.8 tok/s | — |
Convert it with python3 -m tools.convert --recipe gemma4_gguf --components text,mtp,vision; draft
three pays on this model because a wider verification batch costs more weight traffic per accepted
token than it saves.
Changed: the V100 container image
The V100 build is ghcr.io/jimmymax/ninfer-v100:latest (:<version> pins a release, sm70 is an
alias), built from the new Dockerfile.sm70: CUDA 12.8 on Ubuntu 24.04, about 3 GB. The CUDA 13 image
for the RTX 30/40/50 line stays at :sm86-sm120a.
docker run --rm --gpus all -p 8000:8000 --ulimit memlock=-1 \
-v "$PWD/models:/models" ghcr.io/jimmymax/ninfer-v100:latest \
ninfer-serve /models/model.ninfer --host 0.0.0.0 --port 8000 --model-id model \
--max-context 200000 --prefill-chunk 2048 --kv-dtype int8 \
--spec mtp --draft-tokens 4 --lm-head-draft --ngram-draft-tokens 0sm86-sm120a stays the sm_86/sm_120a image, which no V100 can run.
Fixed in the image, before anything else shipped
The first image this release published set ENTRYPOINT ["ninfer-serve"], so a compose command: list -- which names
the program itself, as every ninfer compose file does -- started ninfer-serve ninfer-serve /models/x.ninfer and the server stopped with
ninfer-serve: unknown argument: /workspace/models/x.ninfer
The image now sets no entrypoint, like the other ninfer images, and the commands above name
ninfer-serve first. 1.6.1, latest and sm70 are rebuilt with that fix: the binaries are the same,
and nothing else in this release changed.
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.6.1-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity |
ninfer-serve-sm70.tar.gz |
server binary only |
SHA256SUMS.txt |
checksums for both |
Everything in the v1.6.0 highlights
holds: 27K-token prefill 1.00k tok/s, decode 78 tok/s, v2 artifacts read in place.
Documentation
docs/v100/gemma4-wide-rows-2026-10-16.md records the wide-row work, including the residual
added twice by writing the normalized activations back too early. docs/v100/PLAN.md is the
roadmap with one measurement document per stage.
Apache-2.0, as upstream; model weights keep their own licenses (see LICENSE and the README).