Skip to content

NInfer V100 v1.3.0

Choose a tag to compare

@github-actions github-actions released this 10 Oct 11:41

Prebuilt Tesla V100 (sm_70) binaries, compiled with CUDA 12.8 on Ubuntu 24.04 (glibc 2.39). The runtime hard-requires compute capability 7.0 — other GPUs are rejected at startup.

Highlights

Long-context decoding is 37–47% faster, and the first token arrives about two minutes sooner.

This release rewrites the INT8 small-T decode attention (key-split 8-warp Bc=64 with a byte_perm int8->fp16 dequant) and retunes the D256 flash prompt kernel.

Qwen3.8-27B nvfp4, MTP draft 3, INT8 group-64 KV, --max-context 200000, V100-PCIE-32GB. Decode is the mean of 2 seeds × 1024 generated tokens; first token is the cold prefill of the same prompt.

Prompt tokens Prompt Decode tok/s First token
7,353 code 78.2 → 81.1 (+3.7%) 6.8 s
7,792 Chinese 55.2 → 59.2 (+7.2%) 7.4 s
31,538 code 63.7 → 74.3 (+16.6%) 34.1 → 32.4 s
31,924 Chinese 50.3 → 54.8 (+9.0%) 34.5 → 32.9 s
186,420 code 37.4 → 51.3 (+37.3%) 461.7 → 341.1 s
193,104 Chinese 25.8 → 38.1 (+47.4%) 490.6 → 350.2 s

Gains scale with context because attention's share of decode grows with it. The decode gain is isolated to the new kernel by an in-image A/B switch — NINFER_SM70_ATTN_V2=0 restores the previous kernel — and the first-token gain comes from the prompt retune alone.

Other changes since v1.1.0

  • Fixes the DFlash prefill per-chunk binding and waits for device work before releasing resources (upstream ports; the binding bug could write into the wrong state slot).
  • Defaults to spin-based CUDA synchronization, configurable with NINFER_CUDA_SYNC.
  • Ports the upstream materialization search and adaptive planning-budget rewrite, including the budget oracle and resource-manager tests.
  • Adds a push-triggered CI workflow, the bench/v100/ long-context measurement clients and protocol, and docs/V100-PERF-PLAN.md for continued work.

Verification

  • ninfer_softmax_attention_test passes with the new kernel and with NINFER_SM70_ATTN_V2=0.
  • Runtime mechanism checks pass; the DFlash2 real-model test accepts 20/20.
  • Every benchmark above is paired with a re-run of the previous release on the same host (decode repeats within 1.1%).

Assets

File Contents
ninfer-v100-v1.3.0-linux-x86_64-sm70.tar.gz ninfer CLI + ninfer-serve + ninfer-perplexity, docs, license
ninfer-serve-sm70.tar.gz server binary only, ready for docker/README.md step 1
SHA256SUMS.txt checksums

Docker

docker pull ghcr.io/jimmymax/ninfer-v100:v1.3.0

See docker/README.md for the one-click compose recipe and docs/V100-BUILD.md for runtime flags. Model .ninfer artifacts are downloaded separately — see the README model table.

Full changelog: v1.1.0...v1.3.0