NInfer

Selected checkpoints. Maximum single-GPU inference performance.

NInfer is a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single sm_120a NVIDIA GPU (GeForce RTX 5090 or RTX PRO 6000 Blackwell). It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.

NInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:

Model	Weights	NInfer artifact	Size	SHA-256
Qwen3.6-27B	`groupwise-int`	`qwen3_6_27b.ninfer`	17,495,365,888 bytes (16.29 GiB)	`7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b`
Qwen3.6-27B NVFP4	`nvfp4`	`qwen3_6_27b_nvfp4.ninfer`	18,324,064,000 bytes (17.07 GiB)	`bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a`
Qwen3.8-27B	`groupwise-int`	`qwen3_8_27b.ninfer`	18,210,531,328 bytes (16.96 GiB)	`eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e`
Qwen3.8-27B NVFP4	`nvfp4`	`qwen3_8_27b_nvfp4.ninfer`	21,492,695,040 bytes (20.02 GiB)	`bb3360522a06e136e0367f5703414d26272b7285c8a6ab6194135c17dbd81b32`
Qwen3.6-35B-A3B	`groupwise-int`	`qwen3_6_35b_a3b.ninfer`	22,783,246,080 bytes (21.22 GiB)	`1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2`

Qwen3.6-27B and Qwen3.8-27B each expose two registered weight profiles. The version-2 artifact identity selects the profile without a separate runtime flag; Qwen3.8 uses target key qwen3_8_27b while sharing the 27B execution package. The Qwen3.6 nvfp4 profile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4 kernels for decode. The Qwen3.8 nvfp4 profile preserves its source's mixed allocation: NVFP4 MLP weights in Text layers 0–55 and row-scaled FP8 for the token embedding, attention input/output projections, GDN Q/K/V/Z and output projections, output head, and remaining MLP weights. All four 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI, and serving routes.

Performance

The published measurements cover the three Qwen3.6 artifact profiles and the Qwen3.8-27B NVFP4 profile. The Qwen3.8-27B groupwise-int profile is supported by current NInfer builds but is not yet included in a published benchmark campaign.

Concurrent MTP3 decode

Saturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. MTP acceptance is aggregated over the complete request wave. Each concurrency cell reports decode tok/s / MTP acceptance; profiles should be read independently.

Model profile	C=1 tok/s / accept	C=2 tok/s / accept	C=4 tok/s / accept	C=8 tok/s / accept	C8 / C1
Qwen3.6-27B `groupwise-int`	185.8 / 68.2%	247.0 / 69.0%	309.5 / 68.4%	535.0 / 68.3%	2.88×
Qwen3.6-27B `nvfp4`	202.4 / 69.3%	399.7 / 71.4%	699.7 / 69.3%	1,146.9 / 68.6%	5.67×
Qwen3.6-35B-A3B `groupwise-int`	593.0 / 67.2%	877.7 / 68.2%	1,166.0 / 69.8%	1,313.8 / 67.3%	2.22×
Qwen3.8-27B `nvfp4`	143.8 / 48.9%	267.6 / 48.1%	461.1 / 45.8%	766.6 / 46.0%	5.33×

At C=8, Qwen3.6-35B-A3B reaches 1,313.8 aggregate decode tok/s. Qwen3.6-27B NVFP4 reaches 1,146.9 tok/s and 5.67× its C=1 throughput. Qwen3.8-27B NVFP4 has 45.8–48.9% MTP acceptance, versus 67.2–71.4% across the other measured profiles, so aggregate committed throughput reflects both execution performance and speculative acceptance.

Single-request serving

The single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. Targets and weight profiles are reported independently rather than as cross-target comparisons. Requests were submitted serially to a persistent server. The Qwen3.8-27B NVFP4 MTP0 results use the same dedicated serial corpus runner as the Qwen3.6 profiles; its MTP3 results come from the C=1 point of the fixed concurrent-corpus campaign documented in Performance.

Qwen3.6-35B-A3B

MTP0 at a 7,680-token prompt: 15,544.3 prefill tok/s and 271.1 decode tok/s.
MTP0 at a 260,096-token prompt: 5,157.1 prefill tok/s and 188.2 decode tok/s.
MTP3 long reasoning: 620.3–726.2 decode tok/s with 72.7–82.8% acceptance.
MTP3 structured output: 770.9 decode tok/s, 89.1% acceptance, and 3.67 tokens/round.

Qwen3.6-27B (groupwise-int)

MTP0 at a 7,680-token prompt: 3,218.1 prefill tok/s and 77.6 decode tok/s.
MTP0 at a 260,096-token prompt: 1,614.8 prefill tok/s and 54.8 decode tok/s.
MTP3 long reasoning: 161.9–175.4 decode tok/s with 73.4–78.8% acceptance.
MTP3 structured output: 193.0 decode tok/s, 88.7% acceptance, and 3.66 tokens/round.

Qwen3.6-27B (nvfp4)

MTP0 at a 7,680-token prompt: 11,191.5 prefill tok/s and 86.4 decode tok/s.
MTP0 at a 260,096-token prompt: 2,510.6 prefill tok/s and 59.9 decode tok/s.
MTP3 long reasoning: 213.1–231.0 decode tok/s with 76.3–81.1% acceptance.
MTP3 structured output: 252.2 decode tok/s, 89.8% acceptance, and 3.69 tokens/round.
Against groupwise-int on the same corpus and runtime options: 3.48× the 7,680-token prefill throughput, 1.55× the 260,096-token prefill throughput, and 30–32% higher MTP3 decode throughput.

Qwen3.8-27B (nvfp4)

MTP0 at a 7,680-token prompt: 8,340.4 prefill tok/s and 71.2 decode tok/s.
MTP0 at a 260,096-token prompt: 2,203.1 prefill tok/s and 52.9 decode tok/s.
MTP3 long reasoning: 151.4–195.2 decode tok/s with 56.2–76.0% acceptance.
MTP3 structured output: 219.8 decode tok/s, 90.8% acceptance, and 3.72 tokens/round.

See Performance for the full methodology, variability, reproduction command, and per-fixture results.

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

Model profile	AIME 2025	AIME 2026	GPQA-Diamond	ERQA	RealWorldQA
Qwen3.6-27B groupwise-int	86.67%	93.33%	86.87%	—	—
Qwen3.6-27B NVFP4	93.33%	93.33%	84.34%	—	—
Qwen3.6-35B-A3B groupwise-int	90.00%	90.00%	85.35%	—	—
Qwen3.8-27B groupwise-int	96.67%	96.67%	87.37%	66.25%	82.22%
Qwen3.8-27B NVFP4	96.67%	96.67%	90.40%	66.25%	83.53%

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8-27B rows used temperature 1.0 and presence penalty 0.0. The multimodal columns (ERQA and RealWorldQA) ran with --vision at a 81,920-token context limit; the text columns used a 262,144-token limit except Qwen3.8-27B NVFP4, which needs 252,928 to fit the RTX 5090 after weights.

These are single-sample results under that NInfer evaluation profile, not pass@k. See the model cards and full performance document for correct/total counts and evaluation notes.

Requirements

NInfer currently requires:

64-bit Linux;
an sm_120a NVIDIA GPU: GeForce RTX 5090 or RTX PRO 6000 Blackwell;
NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
CMake 3.28 or newer and a C++20-capable host compiler;
pkg-config;
FFmpeg development libraries: libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7;
libcurl >= 7.85;
Ninja, when using the commands below.

The build rejects CUDA architectures other than 120a. Persistent and cooperative kernels size their launch geometry from the live device's multiprocessor count, so a wider sm_120a part is filled without a rebuild; the published performance figures and the measured schedule-selection tables remain RTX 5090 campaigns. There is no install target or packaged binary distribution; NInfer is run from its source build tree.

Host packages

On Debian/Ubuntu, the host-side requirements above are:

sudo apt install build-essential cmake ninja-build pkg-config \
  libavformat-dev libavcodec-dev libavutil-dev libswscale-dev \
  libcurl4-openssl-dev

build-essential supplies the C++20 host compiler, and ninja-build is required by the -G Ninja commands below. Ninja is not merely a convenience here: the CUDA device-link step is memory-hungry, and the Ninja generator places those links in a serialized cuda_link job pool. Under other generators an unbounded parallel build can exhaust host RAM while linking.

Verify the FFmpeg and libcurl versions satisfy the minimums above, since older distributions ship FFmpeg below the required libavformat >= 60:

pkg-config --modversion libavformat libavcodec libavutil libswscale libcurl

CUDA toolkit

Install CUDA 13.1 or newer from NVIDIA's repositories; the distribution's own nvidia-cuda-toolkit package is usually too old to emit sm_120a. If a second, older nvcc is present on PATH (for example /usr/bin/nvcc from a distribution package), CMake selects it and configuration fails with:

nvcc fatal   : Unsupported gpu architecture 'compute_120a'

Point the build at the correct toolkit explicitly, either by exporting CUDACXX or by putting the toolkit first on PATH:

export CUDACXX=/usr/local/cuda/bin/nvcc     # or: export PATH=/usr/local/cuda/bin:$PATH
nvcc --version                              # confirm release 13.1 or newer

Build

git clone https://github.com/Neroued/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

If configuration reports an unsupported compute_120a, the wrong nvcc was selected; set CUDACXX as shown above and configure again into a clean build directory.

The default configuration builds:

build/apps/ninfer
build/apps/ninfer-serve

Tests, benchmarks, and maintainer tools are excluded from the default build.

Docker

Build the runtime image on a 64-bit Linux host with an sm_120a GPU, a CUDA 13.1-compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

docker build --tag ninfer:local .

Download a model into models/ as described below, then run the HTTP server:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_6_27b.ninfer \
  --host 0.0.0.0

Run the CLI from the same image:

docker run --rm \
  --gpus '"device=0"' \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer /models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-new 256

Download a model

Use the Hugging Face CLI to download one of the registered artifacts:

hf download neroued/Qwen3.6-27B-NInfer \
  qwen3_6_27b.ninfer \
  --local-dir models

# Or the 27B NVFP4 weight variant:
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
  qwen3_6_27b_nvfp4.ninfer \
  --local-dir models

# Or Qwen3.8-27B:
hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

# Or Qwen3.8-27B NVFP4:
hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

# Or:
hf download neroued/Qwen3.6-35B-A3B-NInfer \
  qwen3_6_35b_a3b.ninfer \
  --local-dir models

Current NInfer builds accept only the version-2 artifact container, and all five downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; both Qwen3.8-27B profiles were published directly as version 2. Migrate an older exact local file in place:

python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer

Use the same command with qwen3_6_27b_nvfp4.ninfer or qwen3_6_35b_a3b.ninfer for those artifacts. The migration updates only container metadata; it does not rewrite the weight payload. Alternatively, download the current version-2 file again from its Hugging Face repository.

Each .ninfer file contains the weights and frontend resources needed by NInfer. It is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Each artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded. Vision is also disabled by default, so its weights, Vision scratch phase, and frozen request-transient allocation are omitted. Add --vision to the CLI or server process that must accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is available only for the 35B-A3B target and is text-only.

Run the CLI

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 \
  --max-new 256 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Use --messages FILE instead of --prompt for chat history, images, or videos:

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --vision

Answer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and speculative-decoding statistics are written to stderr. See the CLI guide and committed examples for structured input and runtime options.

Run the HTTP server

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

The public model ID defaults to the artifact's identity.model_id; use --model-id only to publish a deployment-specific alias.

Then send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See HTTP serving.

Capabilities

All three registered model IDs support:

text generation with thinking and non-thinking prompt modes;
image, multi-image, video, and mixed multimodal messages;
chunked prefill and CUDA Graph decode;
startup-bounded small-scale concurrent serving with true batched decode;
MTP speculative decoding with draft windows from one to five;
BF16 and INT8 group-64 KV cache;
model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
compatible-prefix reuse;
OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
prompt-rendered function tools and parsed tool calls.

The 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.

Current limits

Only the five (model_id, weights_id) artifact identities listed above are accepted product identities.
Execution is specialized for one sm_120a GPU and one CUDA device.
One Engine owns one resident model and supports a startup-fixed capacity of 1–16 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
--max-context is the logical ceiling of each sequence and is configurable up to the registered models' native 262,144-token limit. --kv-capacity N explicitly sizes the shared Main Text KV pool for all active and retained sequences, while --kv-capacity auto selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one --max-context worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.
Tool calls are parsed and returned to the client; NInfer does not execute tools.
The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.

Documentation

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.

Name		Name	Last commit message	Last commit date
.codex		.codex
apps		apps
bench		bench
deploy		deploy
docs		docs
eval		eval
examples/cli		examples/cli
include/ninfer		include/ninfer
model-cards		model-cards
src		src
tests		tests
third_party		third_party
tools		tools
.clang-format		.clang-format
.clangd		.clangd
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CMakeLists.txt		CMakeLists.txt
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md

Navigation Menu

Repository files navigation

NInfer

Performance

Concurrent MTP3 decode

Single-request serving

Evaluation

Requirements

Host packages

CUDA toolkit

Build

Docker

Download a model

Run the CLI

Run the HTTP server

Capabilities

Current limits

Documentation

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

NInfer

Performance

Concurrent MTP3 decode

Single-request serving

Evaluation

Requirements

Host packages

CUDA toolkit

Build

Docker

Download a model

Run the CLI

Run the HTTP server

Capabilities

Current limits

Documentation

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages