NInfer-4090 for Windows - 2026.10.05 (prebuilt x64)

Prebuilt Windows x64 binaries of NInfer-4090 for the NVIDIA GeForce RTX 4090 (sm_89). No CUDA toolkit, vcpkg or Visual Studio needed.

What's new since 2026.10.04

Faster Bonsai decode, faster Qwen3.8 prefill, same outputs: quick perplexity is bitwise equal and greedy text identical to 2026.10.04 on every tested prompt.

• Bonsai decode +5 % (MTP 2: 215.8 -> 226.8 tok/s on six mixed prompts). The ternary decode kernel prefetches the next weights into L2, reads its activations contiguously and assigns one row per half-warp on the large weights; decode attention stages its query tile in 16-byte chunks.
• Bonsai now defaults to MTP 3 (start-bonsai-server.bat). With the cheaper 4-token verification it is the faster window: 229.3 tok/s on the six prompts (MTP 2: 226.8), 368 against 308 tok/s on edit-style prompts, 2-3 % faster with n-gram, -2 % to +7 % on prose.
• Qwen3.8 int8 prefill +1.5-2.5 %: the largest int8 GEMMs keep one persistent block per SM that loads the next tile while finishing the current one (pp2048 5,713-5,745 -> 5,825-5,891 tok/s, 64K prompt 15.1 -> 14.9 s).

Measured on the RTX 4090 with no display attached, alternating the two releases; details in docs/maintainer/bonsai-ternary-design.md section 9.1, items 33-36.

	Qwen3.8-27B int8	Ternary Bonsai 2 27B
Prefill pp2048	5,830-5,890 tok/s	6,080 tok/s
64K-token prompt	14.9 s	14.5 s
Decode, MTP 3, six mixed prompts	120 tok/s	229 tok/s
Decode, MTP 3 + n-gram, edit-style prompts	289 tok/s	648 tok/s

Against the official llama.cpp with the same Qwen3.8-27B BF16 source (Q4_K_M): prefill 1.75-2.0x faster, decode 1.1x faster without speculation and 1.2-1.3x with MTP 3 (comparison).

Download and run

1. Download ninfer-4090-windows-x64-2026.10.05.zip below and unzip it.
2. Requirements: RTX 4090, Windows 10/11 x64, NVIDIA driver 595 or newer (tested 595.97), VC++ 2015-2022 x64 redistributable.
3. Download a model (pip install -U huggingface_hub):

   hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8.ninfer --local-dir models
   hf download jgamboa/Ternary-Bonsai-2-27B-NInfer-4090 bonsai2_27b_vl_mtp_q4q5.ninfer --local-dir models
   

4. Run start-qwen38-server.bat or start-bonsai-server.bat; API at http://127.0.0.1:8080/v1, live monitor at /monitor.

The build targets the RTX 4090 only. Other RTX 40-series cards share the architecture and may work with Bonsai (untested); RTX 50-series should use Neroued/ninfer, RTX 30-series Don-Chad/ninfer-3090.

SHA-256 (zip): d7dcd100aadb87d9614764a6186e68b9454e2d2c3087a0c5a0fc66c3ba5196c4

NInfer-4090 for Windows - 2026.10.04 (prebuilt x64)

Prebuilt Windows x64 binaries of NInfer-4090 for the NVIDIA GeForce RTX 4090 (sm_89). No CUDA toolkit, vcpkg or Visual Studio needed.

What's new since 2026.09.27b

Server reliability fixes backported from Neroued/ninfer and sergiuszm/ninfer-4090. Model output and speed are unchanged.

• No more server lock-up in long agent sessions. Under checkpoint-memory pressure with shared prefixes, the server could reject a request with "materialized sequence does not match its active entitlement" and stay unavailable until restarted (sergiuszm/ninfer-4090#9). Admission and capture planning now count only the checkpoints a request owns. A new regression test reproduces the original failure.
• Vision budget is per image. With --vision, --vision-max-tokens (default 8192) used to cap all images in a prompt together, so agents that resend screenshots failed with media_budget_exceeded after ~10 images. It now caps each image; the whole prompt allows up to 32K vision tokens.
• Safe cancellation. Cancelling a request right as it starts waits for the GPU before its buffers are reused.
• DFlash2 long prompts. Each prefill chunk binds its own state slot and KV row (--spec dflash2).
• Anthropic streams: a client disconnect is reported as such instead of a 500. Requests are counted before they are submitted to the engine.

Current figures (RTX 4090 with no display attached, greedy, thinking off):

	Qwen3.8-27B int8	Ternary Bonsai 2 27B
Prefill pp2048	5,730-5,790 tok/s	6,110 tok/s
64K-token prompt	14.8-14.9 s	14.4 s
Decode, MTP, six mixed prompts	120 tok/s	218 tok/s
Decode, MTP + n-gram, edit-style prompts	up to 289 tok/s	up to 532 tok/s

Against the official llama.cpp with the same Qwen3.8-27B BF16 source (Q4_K_M): prefill 1.75-2.0x faster, decode 1.1x faster without speculation and 1.2-1.3x with MTP 3 (comparison).

Download and run

1. Download ninfer-4090-windows-x64-2026.10.04.zip below and unzip it.
2. Requirements: RTX 4090, Windows 10/11 x64, NVIDIA driver 595 or newer (tested 595.97), VC++ 2015-2022 x64 redistributable.
3. Download a model (pip install -U huggingface_hub):

   hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8.ninfer --local-dir models
   hf download jgamboa/Ternary-Bonsai-2-27B-NInfer-4090 bonsai2_27b_vl_mtp_q4q5.ninfer --local-dir models
   

4. Run start-qwen38-server.bat or start-bonsai-server.bat; API at http://127.0.0.1:8080/v1, live monitor at /monitor.

The build targets the RTX 4090 only. Other RTX 40-series cards share the architecture and may work with Bonsai (untested); RTX 50-series should use Neroued/ninfer, RTX 30-series Don-Chad/ninfer-3090.

SHA-256 (zip): fd397d124e3ab195db35962196cf8414d61edd61ea9ede9beddad53eeeac572f

NInfer-4090 for Windows - 2026.09.27b (prebuilt x64)

Prebuilt Windows x64 binaries of NInfer-4090 for the NVIDIA GeForce RTX 4090 (sm_89). No CUDA toolkit, vcpkg or Visual Studio needed.

What's new since 2026.09.27

• Qwen3.8 int8 prefill 6-14 % faster: the A8 GEMMs walk K in 128-column steps (one barrier per two scale groups). 64K-token prompt 18.1 -> 16.7 s, 128K 45.3 -> 42.5 s in an A/B on the same machine.
• Bonsai decode ~3 % faster: decode-width RMSNorm quantization per block, the GDN recurrence loads the next token's inputs ahead, and the round copies no longer use the copy engine (fewer idle gaps on Windows). MTP 2: 187 -> 193 tok/s.
• Outputs are unchanged: identical perplexity and greedy text against the previous release.

Against llama.cpp (same machine, same Qwen3.8-27B BF16 source)

llama.cpp a894dae with a Q4_K_M GGUF (15.65 GiB, MTP layer) against NInfer with the int8-prefill artifact, 8-bit KV, greedy, thinking off:

	llama.cpp	NInfer
Prefill pp512 / pp2048	2,723 / 2,676 tok/s	4,777 / 5,124 tok/s
Prefill 8K / 64K / 128K-token prompt	3.0 / 30.7 / 75.9 s	1.5 / 16.5 / 41.8 s
Decode, no speculation	43.0 tok/s	48.1 tok/s
Decode, MTP 3, six prompts	87.1 tok/s	106.4 tok/s
Decode, MTP 3, 30K-token document	66.8 tok/s	87.2 tok/s

Both engines ran in the same session, about 10 % slower in absolute terms than a cool, idle card; rerun alone, NInfer measured pp512 / pp2048 5,339 / 5,790 tok/s, tg128 54.7 tok/s and the 64K prompt in 14.8 s.

Method: docs/llamacpp-comparison.md.

Download and run

1. Download ninfer-4090-windows-x64-2026.09.27b.zip below and unzip it.
2. Requirements: RTX 4090, Windows 10/11 x64, NVIDIA driver 595 or newer (tested 595.97), VC++ 2015-2022 x64 redistributable.
3. Download a model (pip install -U huggingface_hub):

   hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8.ninfer --local-dir models
   hf download jgamboa/Ternary-Bonsai-2-27B-NInfer-4090 bonsai2_27b_vl_mtp_q4q5.ninfer --local-dir models
   

4. Run start-qwen38-server.bat or start-bonsai-server.bat; API at http://127.0.0.1:8080/v1, live monitor at /monitor.

The build targets the RTX 4090 only. Other RTX 40-series cards share the architecture and may work with Bonsai (untested); RTX 50-series should use Neroued/ninfer, RTX 30-series Don-Chad/ninfer-3090.

SHA-256 (zip): bcc55ed0de830ce5f4bfd18f05320a9890e2a820256aa4cedf1be18f388d6b82

NInfer-4090 for Windows - 2026.09.27 (prebuilt x64)

Prebuilt Windows x64 binaries of NInfer-4090 for the NVIDIA GeForce RTX 4090 (sm_89). No CUDA toolkit, vcpkg or Visual Studio needed.

Download and run

1. Download ninfer-4090-windows-x64-2026.09.27.zip below and unzip it.
2. Requirements: RTX 4090, Windows 10/11 x64, NVIDIA driver 595 or newer (tested 595.97), VC++ 2015-2022 x64 redistributable. The CUDA runtime is built into the executables.
3. Download a model (pip install -U huggingface_hub):

   hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8.ninfer --local-dir models
   hf download jgamboa/Ternary-Bonsai-2-27B-NInfer-4090 bonsai2_27b_vl_mtp_q4q5.ninfer --local-dir models
   

4. Run start-qwen38-server.bat or start-bonsai-server.bat, then use http://127.0.0.1:8080/v1 from any OpenAI- or Anthropic-compatible client. Live monitor: http://127.0.0.1:8080/monitor.

Contents

ninfer.exe (CLI), ninfer-serve.exe (OpenAI/Anthropic-compatible server), ninfer-perplexity.exe, FFmpeg/libcurl/zlib DLLs, two launchers, README.txt and licenses/ (Apache-2.0 for NInfer; FFmpeg 8.1.2 LGPL-2.1, curl, zlib).

Performance (measured on an RTX 4090, Windows 11)

	Qwen3.8-27B int8 prefill	Ternary Bonsai 2 27B
Prefill, 2K tokens (pp2048)	5,138 tok/s	6,027 tok/s
Prefill, 64K / 128K-token prompt	16.5 s / 41.3 s	14.6 s / 37.4 s
Decode, MTP, mixed prompts	~107 tok/s	~188 tok/s
Decode, MTP + n-gram, edit-style prompts	up to 289 tok/s	up to 532 tok/s

The official llama.cpp prefills a Qwen3.8-27B Q4_K_S GGUF at 2,729 tok/s (pp2048) on the same card.

What's in this build (commit 40a9585)

• Qwen3.8 int8 prefill: Q4/Q5 weight codes on int8 tensor cores, same weights and decode as the official artifact.
• Pipelined one-CTA-per-SM GEMMs for Q4/Q5 and ternary weights, including mixed 128/64-token tiles for the 5120-row weights.
• Faster int8 activation quantization (Hadamard rotation in registers) and the Qwen3.8 MLP quantization fused where the activations are produced.
• Faster long-context prompt attention for int8 and E8-lattice KV.
• N-gram drafts chained after MTP, Ternary Bonsai 2 27B support, full 262K context.

Every change keeps outputs bit-identical to the previous build (same perplexity and greedy text) except the opt-in int8 prefill artifact. Details and methods: README, WINDOWS_PORT.md.

Tip: if the 4090 also drives your monitor, decode loses ~15 %; connect the monitor to the motherboard's integrated graphics for full speed.

SHA-256 (zip): a0c44696a59a30be8617bd0378ba0ba303d877ba0435a01a76ac985bcc813df1