Repository navigation
NInfer V100 v1.3.0
Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.
Highlights
Long-context decoding is 37–47% faster, and the first token arrives about two minutes sooner.
This release rewrites the INT8 small-T decode attention (key-split 8-warp Bc=64 with a byte_perm int8->fp16 dequant) and retunes the D256 flash prompt kernel.
Qwen3.8-27B nvfp4, MTP draft 3, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB. Decode is the mean of 2 seeds × 1024 generated tokens; first token is the cold prefill of the same prompt.
| Prompt tokens | Prompt | Decode tok/s | First token |
|---|---|---|---|
| 7,353 | code | 78.2 → 81.1 (+3.7%) | 6.8 s |
| 7,792 | Chinese | 55.2 → 59.2 (+7.2%) | 7.4 s |
| 31,538 | code | 63.7 → 74.3 (+16.6%) | 34.1 → 32.4 s |
| 31,924 | Chinese | 50.3 → 54.8 (+9.0%) | 34.5 → 32.9 s |
| 186,420 | code | 37.4 → 51.3 (+37.3%) | 461.7 → 341.1 s |
| 193,104 | Chinese | 25.8 → 38.1 (+47.4%) | 490.6 → 350.2 s |
Gains scale with context because attention's share of decode grows with it. The decode gain is isolated to the new kernel by an in-image A/B switch — NINFER_SM70_ATTN_V2=0 restores the previous kernel — and the first-token gain comes from the prompt retune alone.
Other changes since v1.1.0
- Fixes the DFlash prefill per-chunk binding and waits for device work before releasing resources (upstream ports; the binding bug could write into the wrong state slot).
- Defaults to spin-based CUDA synchronization, configurable with
NINFER_CUDA_SYNC. - Ports the upstream materialization search and adaptive planning-budget rewrite, including the budget oracle and resource-manager tests.
- Adds a push-triggered CI workflow, the
bench/v100/long-context measurement clients and protocol, anddocs/V100-PERF-PLAN.mdfor continued work.
Verification
ninfer_softmax_attention_testpasses with the new kernel and withNINFER_SM70_ATTN_V2=0.- Runtime mechanism checks pass; the DFlash2 real-model test accepts 20/20.
- Every benchmark above is paired with a re-run of the previous release on the same host (decode repeats within 1.1%).
Assets
| File | Contents |
|---|---|
ninfer-v100-v1.3.0-linux-x86_64-sm70.tar.gz |
ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license |
ninfer-serve-sm70.tar.gz |
server binary only, ready for docker/README.md step 1 |
SHA256SUMS.txt |
checksums |
Docker
docker pull ghcr.io/jimmymax/ninfer-v100:v1.3.0See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table.
Full changelog: v1.1.0...v1.3.0