NInfer Windows v1.2.1 (RTX 5090)

What changed in 1.2.1

A quick-turnaround patch release following v1.2.0, incorporating full upstream synchronization and critical Windows runtime stability fixes:

• Upstream Synchronization (39 commits): Merged all algorithmic and architectural updates from upstream Neroued/ninfer@dev (05327b22..46ee5c3b):
  • Unified RequestStats and EngineSnapshot runtime observability (engine.snapshot().runtime, snapshot().ranks.at(0).memory), retiring superseded publish_kv_capacity.
  • Upstream distributed startup, worker loops, and model preparation.
  • Clamped Expert Parallel (EP) MoE fixtures, Sigmoid routing oracles, and SwiGLU BF16 tests.
  • Upstream observability clocks, host timing refactors, and vocabulary reduction / sampling operations.
• Native Windows Memory Planning Fix: Resolved fatal engine startup failure (cannot read system memory availability) caused by upstream's Linux-only /proc/meminfo query by implementing native Windows GlobalMemoryStatusEx in src/core/device_memory.cpp.
• MSVC 19.44 / 14.51 Template & CTAD Portability: Fixed MSVC class template argument deduction (CTAD) in braced initializer lists across vocabulary_logprobs.cpp and vocabulary.cpp with explicit std::array<int, N> types and included <array>.
• PE Import Dependency Audit: Integrated zero-dependency PE header import analysis into release packaging (package_release.py and check_archive_contents.py) to permanently prevent missing DLL errors at packaging time.
• Hardware Verification: Re-verified all 8 production launchers on RTX 5090 hardware: all 8 lanes load in ~7.5s, commit 262,144 KV tokens, and pass chat completion and image multimodal inference.

Assets & Checksums

• ninfer-windows-v1.2.1-rtx5090.zip (319.0 MB)
• SHA-256: 49e31f2448cf208437a3cbc084d77419219d6e32ff5da6c26e064137039ccab2

NInfer Windows v1.2.0 (RTX 5090)

NInfer Windows v1.2.0 (RTX 5090)

What changed in 1.2.0

This is the first release since v1.1.0. Two drafts of these notes existed — one calling this
release 1.2.0 and a later one 1.3.0 — and they are merged here into 1.2.0, because versions are
published in order and never skipped, and 1.2.0 has never been published. Shipping it as 1.3.0
would leave a 1.1.0 → 1.3.0 gap in the public list, which reads as a withdrawn release. Everything
below is what a user on v1.1.0 gets.

All five artifacts are now built by this port from their published sources rather than fetched from
cometkim, and one draft rule replaces another across every line. All eight launchers change, and two
are new.

• The QUASAR line is rebuilt from its QAT source. Every text linear is imported from
  QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 with its own activation scale, so nothing is re-encoded and no
  corpus is needed. Hashing every binding against the artifact it replaces leaves 1394 of 1513
  byte-identical, so the source repository moving between revisions did not change the weights. The 96
  that differ are gdn/a_projection and gdn/b_projection, which this build takes pristine from the
  base where the fork decoded them from the source's quantized control words — and at (96, 5120) the
  NVFP4 layout cannot hold them at all.
• The unsloth line is rebuilt, its MLP imported and its FP8 attention, linear-attention and MLP
  56-63 encoded from the BF16 base. This source carries no activation scale for those 233 matrices —
  checked exhaustively — so the divisors are measured on this checkpoint by
  tools/convert/calibration.py and committed with it. Borrowing another quantization's stored scales
  instead measured 2.2%/0.45% worse: an activation maximum does not transfer between weight
  realizations. Perplexity on the full corpus at fp8 KV is 4.724219 against the published build's
  4.98768 (docs/perplexity-baseline.md, the shipping gate's own table, re-taken 2026-10-07 and
  reproduced to the last digit on 2026-10-10). The --quick pair recorded on 2026-09-24 was 4.75750
  from 4.82452; --quick is one stream per domain and is not comparable, which that document explains.
• A fourth line joins the release. start_nvidia_v3_dflash2_vision.bat and
  start_nvidia_v3_mtp4_vision.bat serve NVIDIA's own ModelOpt quantization of the base model: its
  NVFP4 MLP imported on all 64 layers, its FP8 attention encoded from the BF16 base using the
  checkpoint's per-site scales. Perplexity is 4.90168 against the official stock's 4.90169 on the
  full corpus — the same, not better — while the file is 20% smaller (18.95 GB against 23.72 GB),
  contains no FP8 tensor where the official has 146. It reaches the full 262,144 at the fp8 KV the
  launchers use, where the copy of the official artifact measured here stops at 240,000 on MTP and lower
  still on DFlash2; the published official does reach 262,144 at int8 KV, so the reach is a property of
  the build and the KV dtype together rather than a wall. The artifact reference carries the exact
  ceilings.
• The draft's projections are encoded as NVFP4 on three lines, where upstream's recipe gives them
  Q8. Measured on the port's own bench: 58.0% against 54.8% acceptance on the QUASAR lane, and 68.8%
  against 49.4% on the unsloth line's DFlash2 lane, which is also 19% faster. It touches no text weight,
  which the MTP digests confirm is bit-identical. It is not applied to Swift, where it lost: 57.7%
  against 60.9% on that artifact's DFlash2 lane. The z-lab draft was trained against the stock model's
  hidden states and this finetune's are not the stock ones, so which encoding matches them is a property
  of the target and is measured per artifact rather than carried across.
• The proposal head is now a record for all eight lanes, and no setting changed. ADR-0005 requires
  the spec route, draft depth and proposal head to be a per-profile decision from a record; the two
  artifacts added after that ADR had none, while all eight lanes passed --lm-head-draft. Measured on the
  served sampling, one lane at a time: every lane confirms the shipped setting, and on several of them the
  full head accepts more and still decodes slower, which is why decode rate rather than acceptance is
  the criterion. One lane's greedy measurement pointed the other way, and reversed when the two arms were
  interleaved on the protocol the lanes actually serve — 329.5/323.6 tok/s with the flag against
  257.2/257.9 without. That reversal is what the record carries: taking the first number would have
  shipped a 25 % regression on that lane.
• Two lane figures had drifted, and one artifact could not be measured at all. Sweeping the
  unchanged artifacts, the acceptance figures recorded on 2026-09-17 no longer reproduce: one
  artifact's DFlash2 lane reads 45.7% against a recorded 62.5%, and its MTP d4 lane 62.4% against
  58.3%. Nothing failed in between — the release check verifies that the launchers, the tables and the
  profiles agree with each other, and they did. Every lane figure in this release is re-measured, and
  compared by interleaving the alternatives in one window, because acceptance reproduces across windows
  and throughput does not.
• A --quick perplexity comparison cannot be quoted as a result. --quick scores one stream per
  domain and full four, so one stream decides the number: the NVIDIA build's zhwiki-00 moves 8.85%
  where the full corpus's four Chinese streams move 1.80% together. Its full-corpus figures are
  identical across two domains won and two lost, and that is what the release says they are.
• --quick figures for the QUASAR artifact do not reproduce either: 4.89741 recorded, 4.94879
  measured. Every row in docs/perplexity-baseline.md now carries the date it was taken.
• A Windows timer defect added up to 16 ms to every request, and is fixed. The engine's scheduling
  poll waits with a 1 ms timeout, which a process that never raises the system timer resolves on the
  15.625 ms tick instead. Measured on one lane, same artifact and same requests: initial_binding was
  bimodal at 0.9 ms or 16.0-17.0 ms, queue_wait bunched at 0-15 ms, and total latency averaged
  77.3 ms. ninfer-serve now calls timeBeginPeriod(1) and the same requests read a flat 2.0-2.6 ms
  initial_binding with 56.3 ms total. Decode and acceptance are unmoved (fingerprint code cell
  320.1 tok/s / 45.9%). ninfer-cli runs the same loop once per invocation, where it is invisible
  beside model load, and ninfer-perplexity does not run it at all.
• The three apps set the console code page, so their output renders as UTF-8 in any console rather
  than through the machine's OEM code page. Verified in both directions with a probe that reads the
  console's output code page: 437 before, 65001 after.
• The A16 SwiGLU decomposition above T=16 is gone, and the fused route covers every extent. Every
  A16 call with T>16 used to project gate/up into a BF16 buffer and apply the nonlinearity to that,
  which measured 8.1e-3 against the profile's own 6.3e-3 tail bound — about 29% over, and identically
  at 17, 32, 64 and 129, so it was the rounding rather than the extent. The route, its silu_mul pass
  and its [34816,T] workspace are deleted. The A16 profile in tests/ops/linear/swiglu/cases_nvfp4.cpp
  now runs 19 extents from 1 to 1024 and passes at every one, with no materialised buffer and no second
  pass. Measured, and it is not a uniform speedup: 3-7% slower at T=17-32, even at 64, 8.5% faster at
  129, and a much tighter tail — the old route's p95 at T=64 was nearly twice its median, where the
  fused route's is within 1%. T=16 is the control: it is the same route in both revisions, and its
  0-2.8% spread is the noise floor. Every shipped lane runs a single lane, so the change is invisible
  to them.

Artifacts

Weights	Artifact	Local size	SHA-256
nvfp4qat (QUASAR)	qwen3_8_27b_nvfp4qat.v3.ninfer	17.65 GiB	42359a09…
nvfp4full (unsloth)	qwen3_8_27b_nvfp4full.v3.ninfer	18.36 GiB	4c1616bc…
nvfp4full_noex (unsloth, DFlash2 lane)	qwen3_8_27b_nvfp4full_noex.v3.ninfer	17.65 GiB	0feb3075…
nvfp4swift15 (Swift 1.5)	qwen3_8_27b_nvfp4swift15.v3.ninfer	17.65 GiB	bdb257f7…
nvfp4nvidia (NVIDIA)	qwen3_8_27b_nvfp4nvidia.v3.ninfer	17.65 GiB	d93a60e8…

Five artifacts across eight launchers. The unsloth line ships twice on purpose: the DFlash2 lane runs
the no-exception image and its MTP lane the BF16-exception one, because re-encoding those nine
exception parents to NVFP4 is what returns the native context to the DFlash2 lane and is worth +11.6
acceptance points there, while the same re-encoding costs the MTP head 21.4 (DFlash2 d7 / MTP d5) so one image cannot serve both routes.

download_model.py fetches the source checkpoints every artifact above is built from, at pinned
revisions, and build_model.py converts them with the converter the archive ships under tools/. No
.ninfer file is downloaded prebuilt, so a fresh install builds the image it runs.

Retained in 1.2.0, from the draft that carried this version

Two retention defects, and the configuration that hid them. All six launchers change, so an upgrade
behaves differently without any action on your part.

• Two Swift lanes join the release. start_swift_v3_dflash2_vision.bat and
  start_swift_v3_mtp4_vision.bat serve UkisAI's Swift finetune as a third artifact. Its ModelOpt
  checkpoint keeps attention and GDN in FP8; importing that left 9 GiB of 8-bit weights and capped
  both lanes below the native context. They are now encoded to NVFP4 from the finetune's BF16
  source, with both W8 endpoints Q8, so no FP8 code word reaches the artifact. Perplexity on the
  fixed corpus improves to 4.68429 from 4.84938 for the same recipe impor...

NInfer Windows RTX 5090 - v1.1.0 (v3 Artifact Line + Four Measured Launchers)

NInfer Windows v1.1.0 (RTX 5090)

First Windows release on the v3 artifact line, with speculative decoding working on
upstream-shaped artifacts, four measured-optimal launchers, and two production bugs fixed
that earlier builds shipped.

Requirements

• NVIDIA GeForce RTX 5090 (32 GB, sm_120a) and a recent driver.
• No CUDA toolkit needed at runtime. The CUDA runtime is linked statically; the engine
  loads from this folder.
• Windows 10/11 x64.

What runs

Launcher	Artifact	Spec	Vision	Context	Decode	Draft accept
start_quasar_v3_dflash2_vision.bat	QUASAR QAT	DFlash2 (7)	yes	262,144	343 tok/s	62.5%
start_quasar_v3_mtp4_vision.bat	QUASAR QAT	MTP (4)	yes	262,144	220 tok/s	58.3%
start_ninfer_v3_dflash2_vision.bat	NVFP4-full	DFlash2 (7)	yes	262,144	345 tok/s	63.7%
start_ninfer_v3_mtp5_vision.bat	NVFP4-full	MTP (5)	yes	262,144	254 tok/s	64.2%

Every number was measured on an RTX 5090 with the exact arguments the launcher passes, in one
interleaved pass; every context ceiling is the highest value the engine accepts for that
configuration -- the next step up is refused, not degraded. Decode varies by up to ~9% between
sessions on a card whose clocks are not pinned, so compare lanes to each other and expect your own
absolute figures to differ.

QUASAR is the recommended profile: our own artifact, at the full 262,144 context, with a DFlash2
lane that measures within noise of the other artifact's (343.4 against 344.6 tok/s, measured
interleaved). ADR-0004 records why the second lane rides a third-party repository with no in-house
fallback. Vision is free on both (the with/without comparison is in
ADR-0004).

Getting a model

download_model.bat fetches the recommended QUASAR QAT artifact and verifies its SHA-256.
The QUASAR profile comes from cometkim/Qwen3.8-27B-nvfp4qat-NInfer and the NVFP4-full
profiles from cometkim/Qwen3.8-27B-nvfp4full-NInfer; both repositories ship a v3 container, so
either downloads and runs directly. Both are SHA-256 verified by download_model.bat, which
offers the choice. The pinned size and hash are the published artifact's: if a download is
refused on size, the repository republished and the pin needs updating from HuggingFace's blob
metadata (a republish is how the previous pin went stale).

Put the .ninfer file at C:\AI\models\ (the path launcher_env.bat expects), then double
click the launcher you want. Each launcher checks the engine and the artifact exist before
starting, and leaves the failure on screen if they do not.

A pre-existing v2 copy must be upgraded first

Neither published repository ships v2 any more, so a fresh download needs no upgrade. A copy
fetched before the republish does: the v3 engine rejects v2 containers outright and names
the tool in the error:

python3 tools/upgrade_ninfer_v2_to_v3.py INPUT.ninfer OUTPUT.ninfer

That upgrade is offline and preserves the weight bytes. On Windows this now works; in
earlier builds the script called POSIX-only os.posix_fadvise/os.fdatasync and died with
AttributeError. The tool ships in this archive, beside the chat_templates/ data it reads.

Speculative decoding is not bit-identical to plain decoding

Greedy output differs between no-spec, each MTP depth and DFlash2, deterministically. This
is documented engine behaviour rather than a defect: acceptance compares a proposal token
against the target argmax for its verify column, and the maintainer notes state that
speculation "does not impose token or logits equality between different quantization,
prefill or kernel paths" — the batched verify kernel is not the single-token decode path,
so a near-tie can flip. Speculation measured 3-4x faster (67-83 tok/s without it, 239-343
with it).

Fixed in this release

• DFlash2 on upstream-shaped artifacts. Our loader demanded a fused
  dflash2/layers/*/attention/query_key_value parameter that upstream's converter never
  emits — it groups attention/query, attention/key and attention/value instead. Any
  artifact not produced by our own upgrade path was refused at startup with
  FATAL missing logical parameter. The runtime already assembles the fused parent itself,
  so the redundant requirement is gone. The official Qwen3.8-27B and community
  fuller-NVFP4 artifacts now serve with --spec dflash2.
• A graph-capture fault in the NVFP4 A4 TMA route. The Windows port kept the tensor-map
  descriptor in a device buffer filled from a caller stack frame; under CUDA Graph capture
  that copy becomes a node whose source is gone by replay, so the TMA unit read a dead
  frame and the kernel trapped with an illegal instruction. The descriptor now travels by
  value in parameter space, as upstream does, which is also what makes it replay-safe.
• Artifact reads blocked writers. FILE_SHARE_READ alone meant Windows refused any
  write to a file the engine had open, where POSIX permits it. Reads now share
  read/write/delete.
• Python tooling could corrupt artifacts on Windows. os.open without O_BINARY opens
  in text mode, which translates CRLF and stops at 0x1A inside payload bytes, and
  os.pread/os.pwrite do not exist there. Both are now portable.
• Context-cache bounds. The launchers set --max-shared-prefixes 7 --max-private-continuations 8 --max-long-anchors-per-continuation 4. With the engine
  defaults, five distinct prompts resent gave 1/5 round-2 hits at a 19.8% token-level hit
  rate, with four of five re-prefilling in full on every call and no signal; the bounds
  give 5/5 at 99.1% and cost nothing measurable.
• CORS removed from every launcher. They passed --cors with no --api-key, which
  lets any browser page on the machine drive the model and read completions. opencode talks
  to these endpoints as a native HTTP client, so CORS bought nothing.

Known limitations

• All four profiles reach the full 262,144 context with Vision. Earlier builds capped the
  NVFP4 lane because that artifact carried 19.7 GiB of device weights; the one shipped now
  carries 17.0 GiB.

Release numbering

Versions are published in order and never skipped. v1.0.1 and v1.0.2 were built on the
maintainer's machine and never released — their archives are still in C:\AI\releases — so the
published list reads 1.0.0, 1.0.3, .... Nothing was withdrawn, and 1.0.0 is unaffected.

Patch numbers carry fixes to the shipped profile set; a minor number carries a new artifact line,
which is what 1.1.0 is: the v3 container, the measured profile table, and the QUASAR lane.

Verification

• Archive ninfer-windows-v1.1.0-rtx5090.zip, 548.4 MB
  SHA-256: ef6f6e1fbdd1747f90a9abd52e2f287e1148d616761c330db0cbc410a57d26ec
• Source suite: 124 of 124, empty known-failure baseline (tools/release/check_test_baseline.py).
• End-to-end: all four launchers started, served a chat completion and accepted an image, each
  reaching the full 262,144 context (tools/release/verify_launchers_v3.py, 4/4). The launcher run
  predates this repackage and the binaries are unchanged by it.
• Rebuilt after publication so the archive carries the corrected README. The archive first
  published as b5d8b38aa13f387ab417f4c3b3c2a809d7fea072145c6cd6b71c271573d9e8a5.

NInfer Windows RTX 5090 - v1.0.6 (Client Protocol Resilience + WDDM VRAM Headroom)

NInfer 5090 Windows - v1.0.6 Release

High-performance native Windows MSVC port of the NInfer engine for NVIDIA GeForce RTX 5090 (sm_120a).

Client Compatibility & Serving Protocol

• Compliant JSON Error Envelopes: All unrendered 404, 405 (method_not_allowed) and generic server errors are now returned as OpenAI/Anthropic JSON error payloads (application/json) with x-request-id, preventing client JSON parser crashes across VS Code, OpenCode Desktop, Cline, Roo Code, and Continue.
• Modern CORS & Distributed Tracing: Added support for Stainless SDK headers (x-stainless-*) and W3C trace context (traceparent, baggage), with a dedicated preflight 204 handler.
• Socket Resilience & TCP Keepalive: Explicit read and write timeouts matching request budgets, eliminating cpp-httplib's default 5-second socket drop during long-running prefill and generation streams. Native Winsock TCP keepalive probes enabled on Windows sockets.
• Model Alias Deduplication: Centralized and exported strip_model_prefix and is_valid_model_id across model listing, retrieval, and completion endpoints.

Windows Runtime & Systems Safety

• Direct I/O Multi-Thread Safety: Allocated dedicated Win32 event handles (CreateEventW) per read operation with RAII EventGuard lifecycle management, eliminating multi-threaded handle race conditions and cleanly detecting EOF.
• WDDM VRAM Headroom Protection: Enforced a 512 MiB minimum automatic VRAM headroom under Windows WDDM drivers when available runtime bytes exceed 1 GiB, protecting against GPU driver paging and system memory spillover.
• Non-FFmpeg Decoders: Added inspect_image and inspect_video stubs in non-FFmpeg build paths to guarantee clean symbol resolution.
• Portable Release Launchers: All startup batch scripts now probe %~dp0 root binaries before falling back to build\apps\, allowing packaged ZIP releases to run standalone without a developer build tree. FFmpeg runtime DLLs are bundled directly into release packaging.

Downloads

Asset	Contents
ninfer-windows-v1.0.6-rtx5090.zip	Standard build — ninfer.exe, ninfer-serve.exe, ninfer-perplexity.exe
ninfer-windows-vision-v1.0.6-rtx5090.zip	Vision build — adds ninfer-serve-vision.exe and the vision launchers

Requirements

• NVIDIA GeForce RTX 5090 (sm_120a) with a current driver. The CUDA runtime (nvcuda.dll, nvcudart_hybrid64.dll) is provided by the display driver, not bundled.
• Model artifacts are not bundled — run download_model.bat, or use a registered .ninfer artifact.

Full details in RELEASE_NOTES.md.

NInfer Windows RTX 5090 - v1.0.5 (DFlash2 Speculative Decoding + MoE Warp Merge)

NInfer 5090 Windows - v1.0.5 Release

High-performance native Windows MSVC port of the NInfer engine for NVIDIA GeForce RTX 5090 (sm_120a).

🚀 Highlights & New Features in v1.0.5

• DFlash2 Speculative Decoding Engine (--spec dflash2 --draft-tokens 7):
  • Added native support for DFlash2 block-diffusion speculative decoding for Qwen3.8-27B.
  • 265.5 tok/s on Code generation (+36.6% speedup over MTP3).
  • 356.8 tok/s on Structured JSON outputs (+62.3% speedup).
  • 321.1 tok/s on long-context math/reasoning benchmarks (+64.5% speedup).
  • Includes dedicated batch launcher: start_ninfer_dflash2.bat.
  • New dedicated CUDA kernels: candidate selector (candidate_selector_path.cu), dynamic grouped conv (w8_dynamic_grouped_conv_add_materialized.cu), fused linear top-k (linear_topk.cpp), and 2048 sliding window attention.
• Sparse MoE Warp Merge Optimization:
  • Transitioned MoE dispatch to one CTA per token in small-T S2 and replaced eight dependent reduction rounds with a single fused bitonic warp merge (perf(sparse_moe)).
• Exact Agent Prefix Reuse:
  • Resolved edge-case token divergence across multi-turn coding agent tool execution loops (OpenCode Desktop, Cursor, Cline), guaranteeing near-100% prefix cache hits and sub-400ms TTFTs on 200k+ token sessions.
• Speculative Terminal Settlement Fix:
  • Preserves KV cache coverage during speculative decoding terminal token settlements.
• Tool Calling & Reasoning Reliability:
  • Preserved structured tool call intent in Qwen tool parser (tool_call_parser.cpp).
  • Preserved Anthropic thinking blocks across server restarts.
  • Added --default-thinking-budget CLI option (configured to 4096 in startup batch files).
• Operational Telemetry:
  • Standardized llama.cpp-style request timing and prompt progress metrics.
• Windows MSVC Compatibility & Automation:
  • Resolved thread-safe time conversion with localtime_s in logging.cpp.
  • Added automated release packager script: package_windows_release.bat.

---

📦 Included Release Assets

1. ninfer-windows-v1.0.5-rtx5090.zip: Standalone Text profile (262k context, FP8/NVFP4/K8V4, MTP5, DFlash2). Includes start_ninfer_5090.bat, start_ninfer_dflash2.bat, and download_model.bat.
2. ninfer-windows-vision-v1.0.5-rtx5090.zip: Standalone Vision profile (131k context, FP8/NVFP4/K8V4, MTP5, DFlash2, FFmpeg shared runtime DLLs). Includes start_ninfer_vision.bat and download_model.bat.

---

⚡ Quick Start

1. Download and extract your chosen ZIP archive.
2. Run download_model.bat to download the qwen3_8_27b_nvfp4.ninfer model weights (~20 GB).
3. Choose your preferred startup script:
   • start_ninfer_5090.bat: Default MTP5 profile (~210–234 tok/s, full 262k context, 100% compatible with base weights).
   • start_ninfer_dflash2.bat: Ultra-fast DFlash2 profile (up to 356 tok/s for models with DFlash2 companion weights).
   • start_ninfer_vision.bat: Vision multimodal profile for image and video analysis (131k context).
4. Connect OpenCode Desktop, Cursor, or any OpenAI-compatible client to http://127.0.0.1:8080/v1 (Model ID: qwen3.8-27b).

---

📄 License & Attribution

Licensed under the Apache License 2.0. Based on upstream Neroued/ninfer.

NInfer Windows RTX 5090 - v1.0.4 (NVFP4/K8V4 KV Cache + Upstream Sync)

NInfer 5090 Windows - v1.0.4 Release

High-performance native Windows MSVC port of the NInfer engine for NVIDIA GeForce RTX 5090 (sm_120a).

🚀 Highlights & New Features in v1.0.4

• New KV Cache Quantization Modes (nvfp4 & k8v4):
  • --kv-dtype nvfp4: Native sub-4-bit FP4 KV cache quantization.
  • --kv-dtype k8v4: Asymmetric 8-bit Key + 4-bit Value hybrid quantization for high precision retention with half value memory.
• Kernel & Execution Optimizations:
  • Optimized RMSNorm Occupancy: Warp scheduling and prefetch unrolling tuned for the RTX 5090 (170 SMs).
  • Fused MoE Gate/Up Staging: Reduced kernel launch latency and optimized token routing.
  • Value-Aware Prefix Cache Re-scheduling: Enhanced anchor validation for recycling long context prefixes.
• Tool Calling & Agent Interoperability:
  • Qwen Tool Call Argument Normalization: Improved JSON escaping and argument extraction for multi-tool calls in agentic environments (OpenCode Desktop, Cline, Cursor).
  • Dynamic Engine Health & Readiness: /health endpoint reports internal pipeline warmup status.
• Unified Operational Logging (spdlog):
  • Fast, thread-safe operational logging with ANSI terminal progress bars during weight loading.
• Network & Dependency Upgrades:
  • Upgraded cpp-httplib to 0.54.1 with enhanced transport resilience.
• Windows MSVC Compatibility:
  • Full native MSVC 2022/2026 + CUDA 13.3 support with UTF-8 compiler conformance and Win32 console screen buffer discovery.

---

📦 Included Release Assets

1. ninfer-windows-v1.0.4-rtx5090.zip: Standalone Text profile (262k context, FP8/NVFP4/K8V4, MTP5). Includes start_ninfer_5090.bat and download_model.bat.
2. ninfer-windows-vision-v1.0.4-rtx5090.zip: Standalone Vision profile (131k context, FP8/NVFP4/K8V4, MTP5, FFmpeg shared runtime DLLs). Includes start_ninfer_vision.bat and download_model.bat.

---

⚡ Quick Start

1. Download and extract your chosen ZIP archive.
2. Run download_model.bat to download the qwen3_8_27b_nvfp4.ninfer model weights (~20 GB).
3. Run start_ninfer_5090.bat (or start_ninfer_vision.bat) to launch the local server at http://127.0.0.1:8080/v1 (Model ID: qwen3.8-27b).

---

📄 License & Attribution

Licensed under the Apache License 2.0. Based on upstream Neroued/ninfer.

NInfer Windows RTX 5090 - v1.0.3 (Upstream Merge + FP8 MTP5)

NInfer 5090 Windows - v1.0.3 Release

High-performance native Windows MSVC port of the NInfer engine for NVIDIA GeForce RTX 5090 (sm_120a).

🚀 Highlights & Performance Upgrades

• Full Upstream Merge (d9dbe1ce):
  • Fused TMA SwiGLU: Registered for all NVFP4 block widths on RTX 5090.
  • W8 Vectorized Linear Decode: Widened to 8 codes/thread for doubled decompress throughput.
  • Value-Aware Prefix Scheduling: Keeps frequent conversation prefixes warm in VRAM.
  • Streaming Request Cancellation: Immediately halts GPU compute on client disconnects.
• FP8 KV Cache + MTP5:
  • 207 – 220.8 tok/s generation throughput (~80% MTP5 acceptance rate).
  • 160 – 194.2 tok/s sustained throughput on massive deep-context prompts (tested up to 182k+ tokens).
  • Full 262,144-token context support in 32GB VRAM with ~1.2 GiB headroom.
• 16GB DDR5 Host Offload:
  • Seamless multi-turn session caching via --host-kv-mib 16384 and --host-state-slots 16.
• Multimodal / Vision Support:
  • Vision profile (131k context) with pre-packaged FFmpeg GPL-shared libraries.
• 100% Native Windows MSVC:
  • No WSL2 required. Fully compiled with MSVC 2026/2022 and CUDA 13.3.

---

📦 Included Release Assets

1. ninfer-windows-v1.0.3-rtx5090.zip: Standalone Text profile (262k context, FP8, MTP5).
2. ninfer-windows-v1.0.3-rtx5090-vision.zip: Standalone Vision profile (131k context, FP8, MTP5, FFmpeg DLLs).

---

⚡ Quick Start

1. Download and extract your chosen ZIP archive.
2. Run download_model.bat to download the ~20GB qwen3_8_27b_nvfp4.ninfer model file.
3. Run start_ninfer_5090.bat (or start_ninfer_vision.bat) to launch the local OpenAI-/Anthropic-compatible server at http://127.0.0.1:8080/v1 (Model ID: qwen3.8-27b).

---

📄 License & Attribution

Licensed under the Apache License 2.0. Based on upstream Neroued/ninfer.

NInfer Windows RTX 5090 - v1.0.0

Initial RTX 5090 Release (Native Windows Port)

Initial release of the high-performance Windows MSVC port of the NInfer engine for NVIDIA GeForce RTX 5090 (sm_120a).

Available Builds

1. ninfer-windows-v1.0.0-rtx5090.zip (Text-Only)

   • Lightweight, native MSVC text-only inference engine.
   • Context Capacity: Up to 262,144 tokens with INT8/FP8 KV caching.
   • Standalone runtime with zero external dependencies.

2. ninfer-windows-v1.0.0-rtx5090-vision.zip (Multimodal / Vision)

   • Includes ninfer-serve-vision.exe bundled with pre-packaged FFmpeg GPL-shared libraries.
   • Context Capacity: Configured for multimodal Vision workflows.
   • Supports image and video inputs.

---

Quick Start

1. Download and extract your preferred ZIP archive.
2. Run download_model.bat to download the ~20GB Qwen3.8-27B NVFP4 model artifact from Hugging Face.
3. Launch the server using start_ninfer_5090.bat (or start_ninfer_vision.bat).

Note

For the latest performance optimizations (FP8 KV Cache, MTP5 220+ tok/s, 16GB DDR5 Host Offload, and upstream kernel merges), please see the v1.0.3 Release.