Skip to content

Releases

NInfer-3090 v0.16.1 - community engine adoption and native Windows build

Choose a tag to compare

@Don-Chad Don-Chad released this 16 Oct 20:14

NInfer-3090 v0.16.1 — community engine adoption, native Windows build

This release brings the continued RTX 3090 work from ashalliants' fork back into Don-Chad's
repository, with a downloadable native Windows build.

The source includes Qwen3.8-27B DFlash2, Qwen3.6, and Qwen3.8-Flash-Next Windows preview,
specialized SM86 kernels, ReplaySSM, quantized paged KV, prefix reuse, constrained output, and
OpenAI/Anthropic serving. The release contains Windows x64 binaries only. Linux source and the
community's existing Linux builds remain available separately.

Upgrade from the previous Don-Chad release

  • Model container v3 is required. Old v2 files are rejected. Upgrade a supported v2 file with
    tools/upgrade_ninfer_v2_to_v3.py or use the pinned model downloader. Preserve your original.
  • This is a broad engine adoption, not a drop-in patch to v0.6.1. Recheck existing launch
    commands against the bundled --help and current serving reference. Memory, scheduling and
    metrics contracts have changed since that release.
  • The old custom Anthropic top-level timings extension is retired. The standard Anthropic
    response remains supported; use current logs/metrics for engine observations.
  • CUDA 13.2 Windows build. A compatible NVIDIA driver is required. Qualification used driver
    610.88. CUDA Toolkit installation is unnecessary: the archive includes the required cuBLAS DLLs.
  • Default 27B profile uses DFlash2 with one active lane, quantized rk4v4 KV, cuBLAS prefill
    and vision overlay. The launcher fits its context ceiling to free GPU memory. Several requests
    require a compatible multi-lane profile; this is not high-concurrency datacenter batching.
  • The downloader may fetch an optional private graft using a local Hugging Face login.
    Set NINFER_GRAFTS=off to skip it. Public model inference does not require that graft.

What is included

  • ninfer-serve.exe: streaming OpenAI Chat Completions/Responses and Anthropic Messages server.
  • ninfer.exe: native CLI; ninfer_bench.exe: Engine-route benchmark.
  • run.bat, pinned model downloader, runtime DLLs, release notes and dependency licenses.
  • BUILD_INFO.json records source commit, CUDA toolkit version and SM86 GPU target.
  • Inner SHA-256 checksums and an outer archive checksum file.

Models are downloaded separately. The recommended Qwen3.8-27B artifact is about 19 GB.
The server executable is about 577 MB before compression; this is not a tiny single-file package.

Windows fixes in this release

  • Resolve MSVC nested generic-lambda constant-capture failures in GDN gating and INT8 small-T
    attention without changing the intended kernel mathematics.
  • Use the CUDA 13 graph-edge query signature in the stage-link test while retaining CUDA 12 support.
  • Calculator checks accept both Windows CRLF and Linux LF line endings.
  • Package the cuBLAS version actually linked by the build, including CUDA 13's bin/x64 layout,
    rather than assuming CUDA 12 or trusting the caller's CUDA_PATH.
  • Retain contributor history, original attribution and third-party license notices in the archive.

Measurements and validation

The same runtime source was qualified on one desktop-attached RTX 3090 with CUDA 13.2/MSVC 14.43.
Qwen3.8-27B, DFlash2 K=7, one lane, 16K context cap, rk4v4 KV, thinking and prefix reuse off:
three warmups and six measured requests yielded 116.7 tokens/s median decode and
127 ms median first streamed content, with 52 prompt tokens and 512 generated tokens per
request. Cold startup was about 42 seconds and excluded. This is not a matched comparison with
v0.6.1, another engine, or the fork's separate 140-token/s workload.

The source qualification passed the full native app/test build, 15 selected C++ host tests
(one additional S3 test skipped), 22 Python checks, calculator checks, and Linux script fixtures.
Release packaging additionally requires all bundled programs to start with only Windows on PATH,
fresh-extraction checksums, and a real-card request through the packaged launcher.

No Linux CUDA runtime qualification, full GPU numerical suite, sanitizer campaign or complete
dependency CVE audit is claimed. The targeted source security review found no obvious malicious
behavior in the inspected build/download, process, network and persistence surfaces.
Distinct-GPU execution and Flash-Next were not part of our local inference benchmark.

Credits

Original engine: Neroued. RTX 3090 adaptation: Don-Chad. Substantial continued development
and integration: ashalliants, WarlaxZ, and their contributors. mgscreativa contributed
fixes and proposed the import in PR #30. Flash-Next also adapts attributed work from
wallawalla47/Infernix. Both histories, source notices and CONTRIBUTORS.md are preserved.

The fork import is pinned at 931bd49ba332f48fad4b5230eec5a95ec5701290; the adoption merge is
1418aca37b253549e29f0e77fdb72db9d2b32195. Historical optional Git LFS graft objects could not be
uploaded to this public fork; the current source tree has no Git LFS dependency.

Verified release archive

  • Windows ZIP: 1,190.37 MB; all 42 inner file hashes and the outer SHA-256 verified.
  • Fresh extraction: all three executables passed --help with only Windows on PATH.
  • Packaged run.bat qwen38-27b served a real chat completion on the RTX 3090.
  • The default launcher fitted its 188,416-token ceiling to 174,080 tokens beside the desktop; this checks startup and generation, not long-context quality.
  • Source commit: 8d509ae. CUDA 13.2.1; SM86.
  • The test server was stopped afterward. No Linux binary is included in this release.

NInfer-3090 v0.6.1 - Windows and Linux

Choose a tag to compare

@Don-Chad Don-Chad released this 25 Aug 21:25

NInfer-3090 v0.6.1

This release ships native Windows and Linux x64 binaries for the RTX 3090, with
the current Qwen3.8-27B artifact and the same explicit SM86 runtime profile on
both platforms.

Changes

  • Adds the upstream swscale destination-alignment fix for JPEG decoding, including
    its 300x200 regression fixture.
  • Renders Anthropic and Claude Code system reminders in place after a conversation
    begins, preserving prefix reuse across turns.
  • Streams tool-call whitespace without repeatedly scanning the buffered prefix.
  • Ships Windows ZIP and Linux tarball packages with their platform launch helpers.

Contributors

Validation

The release build and smoke gate uses these components:

  • Windows: Visual Studio 2022 and CUDA 12.8
  • Linux: Ubuntu 22.04, CUDA Toolkit 12.8, GCC 11, CMake 3.28, Ninja, and vcpkg FFmpeg

Each packaged ninfer binary completes a real RTX 3090 Qwen3.8-27B generation before publication.

Limits

  • This release is a correctness and distribution update, not a new throughput claim.
  • Keep the GPU idle while loading the 27B model; a 24 GB RTX 3090 needs the
    documented explicit KV settings for higher-concurrency profiles.

NInfer-3090 v0.6.0 - ReplaySSM and Qwen3.8 reasoning

Choose a tag to compare

@Don-Chad Don-Chad released this 14 Aug 20:34

NInfer-3090 v0.6.0

This release brings ReplaySSM, C8/MTP3, reasoning-effort control, and the official Qwen3.8-27B
groupwise NInfer artifact to a single 24 GB RTX 3090.

Highlights

  • registers qwen3.8-27b/groupwise-int as target qwen3_8_27b;
  • binds its W8 token embedding and output head while reusing the qualified 27B execution package;
  • validates text generation, MTP2/MTP3, paged INT8 KV, CUDA Graphs, and concurrent serving;
  • uses ReplaySSM to reduce speculative recurrent-state memory;
  • validates C1-C4 at 4K and C8 at 8K with MTP3;
  • supports Qwen3.8 low, medium, and xhigh reasoning effort;
  • includes reproducible Qwen3.8 concurrency and MTP sweep tools.

Recommended C8 command

.\ninfer-serve.exe models\qwen3_8_27b.ninfer `
  --max-context 8192 --kv-capacity 8192 --max-concurrency 8 `
  --prefill-chunk 1024 --kv-dtype int8 `
  --spec mtp --draft-tokens 3 --lm-head-draft

C8/8K MTP3 measured 114.73 aggregate end-to-end tok/s with an 8,192-token shared KV pool and
21,818 MiB peak VRAM. A 65,536-token pool also fits, measuring 114.88 tok/s and 23,745 MiB peak.

The official artifact SHA-256 used for validation is
eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e.

Limitations

  • Qwen3.8 validation in this release is text-only.
  • Greedy outputs from ordinary, MTP2, and MTP3 execution were coherent but not byte-identical on
    every benchmark prompt; the performance sweep is not a claim of exact quality parity.
  • Paged KV supports BF16 and INT8. RotorQuant/KV4 is not exposed by this runtime.
  • NVFP4 and TMA remain unavailable on SM86.

NInfer-3090 v0.5.0

Choose a tag to compare

@Don-Chad Don-Chad released this 14 Aug 18:54

NInfer-3090 v0.5.0

This release adds native Windows/SM86 execution for the official Qwen3.8-27B groupwise NInfer
artifact on a single 24 GB RTX 3090.

Highlights

  • registers qwen3.8-27b/groupwise-int as target qwen3_8_27b;
  • binds its W8 token embedding and output head while reusing the qualified 27B execution package;
  • validates text generation, MTP2/MTP3, paged INT8 KV, CUDA Graphs, and concurrent serving;
  • validates C1-C4 at 4K and C8 at 8K with MTP2;
  • includes reproducible Qwen3.8 concurrency and MTP sweep tools.

Recommended C8 command

.\ninfer-serve.exe models\qwen3_8_27b.ninfer `
  --max-context 8192 --kv-capacity 8192 --max-concurrency 8 `
  --prefill-chunk 1024 --kv-dtype int8 `
  --spec mtp --draft-tokens 2 --lm-head-draft

C8/8K MTP2 leaves limited memory margin: 774 MiB physically free and 388 MiB planned slack on the
test system. MTP3 does not fit with the optimized proposal head. Do not run competing GPU workloads.

The official artifact SHA-256 used for validation is
eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e.

Limitations

  • Qwen3.8 validation in this release is text-only.
  • Greedy outputs from ordinary, MTP2, and MTP3 execution were coherent but not byte-identical on
    every benchmark prompt; the performance sweep is not a claim of exact quality parity.
  • Paged KV supports BF16 and INT8. RotorQuant/KV4 is not exposed by this runtime.
  • NVFP4 and TMA remain unavailable on SM86.

NInfer-3090 v0.4.0

Choose a tag to compare

@Don-Chad Don-Chad released this 13 Aug 07:46

Native Windows RTX 3090 release adding upstream paged KV, concurrent MTP execution, compatible-prefix reuse, bounded admission, and improved OpenAI/Anthropic serving.

Measured with the compact 20.84 GiB Qwen3.6-35B-A3B artifact and 4K INT8 KV:

  • C1: 162.7 aggregate tok/s, 22,427 MiB
  • C2: 267.9 aggregate tok/s, 22,743 MiB
  • C4: 366.2 aggregate tok/s, 23,377 MiB
  • C6: 383.4 aggregate tok/s, 24,038 MiB
  • Longer C2 run: 399.1 aggregate tok/s
  • Prefix reuse: 24/26 tokens reused; prefill 371 ms to 10 ms

DFlash is not recommended on RTX 3090. The compact artifact works without DFlash weights; selecting DFlash reports an explicit error.

Validation: native Release build succeeded and 11/11 focused artifact, scheduler, KV/prefix, speculative, and SM86 W8 tests passed.

The Windows archive contains the CLI, server, benchmark, runtime DLLs, guide, license, version, and internal SHA-256 manifest. Model weights are not included.

NInfer RTX 3090 v0.3.1 - 300-400+ tok/s and long-context KV modes

Choose a tag to compare

@Don-Chad Don-Chad released this 30 Jul 11:45

NInfer RTX 3090 v0.3.1 — 35B-class inference at 300–400+ tok/s

NInfer v0.3.1 turns one RTX 3090 into a highly specialized local inference engine for
Qwen3.6-35B-A3B. The entire 20.84 GiB quantized model, INT8 KV cache, CUDA Graphs, and speculative
runtime fit on the 24 GB card.

This release is built for useful local work rather than a single synthetic headline:

Workload Result
Repetition-rich Python repository edit, best clean run 406.44 ± 0.79 output tok/s
Same 1,500-input/1,500-output fixture, release rerun 399.16 ± 2.33 output tok/s
Realistic two-copy code-refactor prompt, 1,500 input + 1,500 output 310.04 ± 0.95 output tok/s
Canonical tg128, MTP-2 262.05 ± 1.62 output tok/s
Canonical tg128, no speculation 187.24 ± 0.27 output tok/s

The 310 tok/s result matters: its prompt contains only two copies of the code rather than the
heavy repetition of the 400 tok/s showcase. Dynamic mode correctly keeps that workload on MTP-3.
The 399–406 tok/s result demonstrates the additional headroom available for repository edits,
templated code, regeneration, and other workloads with reusable local continuations.

What we added

  • Adaptive hybrid speculation. A one-time rolling-hash scan classifies long inputs in under
    0.1 ms. Ordinary requests remain on MTP-3; sufficiently repetitive requests activate target-only
    prompt lookup.
  • Adaptive K15/K8/K5 lookup windows. The runtime verifies the longest useful continuation,
    then falls back to smaller captured windows when the available match is shorter.
  • Lazy MTP realignment. After lookup misses, MTP state is restored only when needed, reducing
    ordinary fallback work.
  • GA102 kernel work. RTX 3090 schedules cover MoE decode, Gated DeltaNet projection and
    recurrent-state work, output/argmax operations, and graph overhead.
  • Cross-platform graph memory planning. Windows and CUDA 13.0 Linux graph-capture footprints
    are both validated within the 24 GB card.
  • Rotated compressed KV caches. New int4, k8v4, and quality-first rk8v4 modes extend
    practical context capacity on a 24 GB RTX 3090. The asymmetric K8/V4 route preserves keys at
    higher precision; rk8v4 additionally rotates values before quantization and reverses the
    transform after attention.
  • Native release packages. Windows 11 and Ubuntu 24.04/WSL2 builds include the CLI,
    OpenAI-/Anthropic-compatible server, benchmark runner, runtime dependencies, and checksums.

Recommended flags

For general 35B-A3B text inference on a 24 GB RTX 3090, start with adaptive hybrid mode:

.\ninfer.exe models\qwen3_6_35b_a3b.ninfer `
  --prompt "Refactor this Python module and return the complete implementation." `
  --max-context 4096 --prefill-chunk 128 --max-new 1500 `
  --kv-dtype int8 --text-only `
  --mtp-draft-tokens 3 --lm-head-draft `
  --prompt-lookup-tokens 15 --prompt-lookup-min-match 4 `
  --prompt-lookup-auto --prompt-lookup-min-context 1000

The same flags work with ninfer-serve:

.\ninfer-serve.exe models\qwen3_6_35b_a3b.ninfer `
  --host 127.0.0.1 --port 8080 `
  --max-context 4096 --prefill-chunk 128 `
  --kv-dtype int8 --text-only `
  --mtp-draft-tokens 3 --lm-head-draft `
  --prompt-lookup-tokens 15 --prompt-lookup-min-match 4 `
  --prompt-lookup-auto --prompt-lookup-min-context 1000

What the flags do:

  • --mtp-draft-tokens 3 enables the model's trained MTP proposal layer with three draft positions.
  • --lm-head-draft selects the optimized proposal head required by the fast MTP path.
  • --prompt-lookup-tokens 15 makes up to 15 tokens from a matching local continuation available
    for target-model verification. The adaptive runtime can use captured K15, K8, or K5 windows.
  • --prompt-lookup-min-match 4 requires at least a four-token repeated suffix before lookup can
    propose a continuation.
  • --prompt-lookup-auto keeps prompt lookup off for inputs that do not contain enough reusable
    structure. This is the recommended general-purpose setting.
  • --prompt-lookup-min-context 1000 delays automatic classification until the input is long enough
    for repetition coverage to be meaningful.
  • --kv-dtype int8 is the supported memory-efficient KV configuration for 35B-A3B on a 24 GB card.
  • --text-only avoids the vision-workspace reservation and is required for the 35B-A3B release
    configuration on an RTX 3090.

For ordinary prose or short prompts, MTP-2 is the conservative controlled-performance mode:

--kv-dtype int8 --text-only --mtp-draft-tokens 2 --lm-head-draft

Omitting --prompt-lookup-auto forces lookup attempts and should be reserved for workloads known
to contain substantial repetition. K15 is not free: the target model verifies 16 positions, so
forced lookup can be slower on non-repetitive text.

Long-context KV modes

The default --kv-dtype int8 remains the conservative quality and performance choice. When a
larger context allocation is required:

  • --kv-dtype k8v4 uses rotated INT8 keys and packed INT4 values. It is the recommended
    long-context capacity mode because it keeps the fast value path.
  • --kv-dtype rk8v4 adds a full value rotation and inverse attention-output rotation. It is the
    quality-first K8/V4 mode, but its extra work makes very large prefills slower.
  • --kv-dtype int4 packs both keys and values and provides the largest cache capacity, with the
    greatest quantization trade-off.

Maximum context is a memory allocation limit, not the attention length paid by every token. A
54K maximum remains fast while only about 4K tokens are populated. Decode slows as the active KV
cache grows because dense attention reads the populated history; on the 65,536-token fixture,
K8/V4 decode measured about 112 tok/s on one RTX 3090.

Model scope

The published RTX 3090 release, its packages, tuning, memory limits, and headline measurements are
built and validated around Qwen3.6-35B-A3B. The 35B-A3B .ninfer artifact is downloaded
separately and is the recommended model for these binaries.

The open-source tree also contains the complete Qwen3.6-27B target, converter, reference
implementation, tests, and build integration. Developers can build the project from source and run
the 27B .ninfer artifact with the same CLI or server. The 27B path has less of the new
35B-A3B-specific GA102 acceleration and is not covered by the 300–400+ tok/s release claims.

Latency

Warm resident time to first token, including prompt execution through the first generated token:

Input TTFT
128 tokens 62.16 ms
512 tokens 242.25 ms
1,500 tokens 737.41 ms

How the numbers were measured

All throughput figures are output tokens divided by decode time, measured on one RTX 3090 using the
same Qwen3.6-35B-A3B artifact, CUDA Graphs, INT8 group-64 KV, text-only mode, five measured
repetitions after warm-up, and no competing GPU workload. Model loading and graph construction are
excluded. The long code tests generate a fixed 1,500 output tokens; tg128 is a shorter controlled
kernel/scheduling comparison.

The 400 tok/s workload is deliberately labeled repetition-rich and is not a universal model-speed
claim. The 310 tok/s two-copy code workload and canonical figures provide more conservative points
of comparison.

Download the 20.84 GiB model separately from
Hugging Face. See the repository README
for exact commands, inference modes, server usage, memory requirements, and full caveats.

This RTX 3090 edition began as a port of Neroued's RTX 5090 NInfer implementation and has since
grown its own GA102 kernels, memory profile, Windows/Ubuntu release workflow, and adaptive
speculative runtime.

NInfer RTX 3090 v0.2.0-v2 build 1 - Windows and Ubuntu

Choose a tag to compare

@Don-Chad Don-Chad released this 22 Jul 07:11

Build 1 of runtime 0.2.0-rtx3090-v2, publishing the finalized source and packages for the RTX 3090 (sm_86) port of NInfer.

Source commit: 464e2d6

Highlights

  • updated 35B-A3B normal and speculative decode schedules
  • controlled MTP-2 result: 265.59 +/- 0.71 tok/s
  • real streamed MTP-3 generation: 284.75 tok/s
  • normal 35B-A3B v2 result: 188.61 +/- 0.30 tok/s
  • native Windows CLI/server support and Ubuntu 24.04/WSL2 support
  • text-only mode allows the unchanged 20.84 GiB 35B-A3B artifact to fit on a 24 GiB RTX 3090

Assets

  • ninfer-rtx3090-windows-x64-0.2.0-rtx3090-v2.zip
  • ninfer-rtx3090-linux-x64-0.2.0-rtx3090-v2.tar.gz
  • SHA256SUMS.txt

Each platform archive includes the CLI, server, benchmark runner, VERSION, README, license, and per-file checksums. Model artifacts are downloaded separately.

Requirements

Windows prebuilt

  • NVIDIA GeForce RTX 3090
  • Windows 11 x64
  • current NVIDIA driver compatible with CUDA 13.x
  • Microsoft Visual C++ 2022 runtime
  • the full CUDA Toolkit is not required for the prebuilt Windows archive; its CUDA runtime is linked statically

Ubuntu prebuilt

  • Ubuntu 24.04 or Ubuntu 24.04 under WSL2
  • NVIDIA GeForce RTX 3090 and a compatible NVIDIA driver
  • CUDA 12 runtime (libcudart.so.12), FFmpeg 6, curl, and the standard C++ runtime

Building the current source on either platform requires CUDA Toolkit 13.0 or newer.

Qwen3.6-35B-A3B requires --text-only on the RTX 3090.

Validation

  • Windows: 60/60 registered tests passed
  • Ubuntu/WSL2: 60/60 registered tests passed
  • on both platforms, two real-model integration tests skip normally when their model paths are not configured
  • real 35B Windows and WSL2 inference completed successfully

Archive SHA-256

  • Windows: e5b4c63abe728ec5d9aa81a8abca728d0f3e53322e855c8ed688ca39e84d2e70
  • Ubuntu: ef81607119f0a6d7156e8d831eccfd1e357b81c83596539070080a14f9c4cd63

NInfer RTX 3090 v0.2.0-rtx3090-v2

Choose a tag to compare

@Don-Chad Don-Chad released this 22 Jul 07:05

NInfer RTX 3090 v2

This release publishes the current RTX 3090 port as both tagged source code and a ready-to-run native Windows x64 package.

Highlights

  • Runtime revision 0.2.0-rtx3090-v2 embedded in binaries and the package.
  • Qwen3.6-35B-A3B text-only inference on a 24 GiB RTX 3090.
  • Latest real streamed MTP-3 result: 284.75 tok/s.
  • Latest repeated controlled MTP-2 result: 265.59 +/- 0.71 tok/s.
  • Native Windows 11 build with required FFmpeg, curl, and zlib DLLs included.
  • CLI, OpenAI/Anthropic-compatible server, and benchmark executable included.

Validation

  • Windows Release build completed successfully with CUDA 13.2 and MSVC 19.44.
  • 60/60 registered Windows tests passed.
  • Two model-artifact-gated integration tests skipped normally.

Model artifacts are not bundled. Download a supported .ninfer artifact using the links in the repository README. Qwen3.6-35B-A3B requires --text-only on a 24 GiB RTX 3090.

Windows ZIP SHA-256: 200b1b5e5bf8edff1050f25351a6eba0c8d84fd0ffba5bf4635668ee2da5355d

NInfer RTX 3090 - Windows x64 - 2026-07-21

Choose a tag to compare

Native Windows x64 release of the RTX 3090 (sm_86) port of NInfer.

Included:

  • ninfer.exe CLI
  • ninfer-serve.exe OpenAI/Anthropic-compatible server
  • ninfer_bench.exe benchmark runner
  • required FFmpeg, curl, and zlib DLLs
  • Windows README, license, and per-file checksums

Requirements:

  • NVIDIA GeForce RTX 3090
  • Windows 11 x64
  • compatible NVIDIA driver
  • Microsoft Visual C++ 2022 runtime
  • model artifact downloaded separately (not included)

Qwen3.6-35B-A3B requires --text-only on the RTX 3090.

ZIP SHA-256: 088968931a118d0a4183553fa861fd18da68ff2b044aea07c6369557642457a2

Validated on native Windows: 60/60 registered tests passed; the two artifact-gated integration tests skip when their model paths are not configured. Real 35B Windows inference completed successfully.