NInfer Windows 0.4.0 (portable, CUDA 13.1)

Live compressed-KV cache quantization (community contribution, PR #1): four new --kv-dtype cache modes — rk8v4, rk4v4, rk4v4-e8, and rk2v4-e8. The E8 modes project keys onto the Conway-Sloane E8 lattice so the K cache runs at 2–4 bits per dimension, making long-context sessions that a full-precision KV cache cannot hold (200k+ context on a 32 GB card) practical. Bundled launchers still default to the full-precision cache; add --kv-dtype to opt in.

OpenAI chat-completions tool messages now accept array content (a mix of text and image_url parts), as allowed by the spec. Tool results carrying a screenshot or other media are no longer rejected with HTTP 400; the media is rendered through the vision pipeline like any other image part. Plain string tool content behaves as before.

Self-contained portable zip: exes, runtime DLLs, per-model launchers (150k baseline), models/ folder, README.txt, SHA256SUMS. Requires Windows 11 x64, RTX 5090 (sm_120a), and a CUDA 13.1-capable NVIDIA driver.

NInfer Windows 0.3.0 (portable, CUDA 13.1)

Synced 8 upstream commits (improved media/vision pipeline, gdn snapshot isolation, Qwen3.8 performance docs)

Added context-window reporting: /v1/models and /v1/models/{id} now include meta.n_ctx = the served --max-context ceiling, so clients that auto-detect context need no manual configuration.

All bundled launchers now set a distinct public model id via --model-id so the groupwise-int and NVFP4 builds of the same base model are distinguishable by clients.

Self-contained portable zip: exes, runtime DLLs, per-model launchers (150k baseline), models/ folder, README.txt, SHA256SUMS. Requires Windows 11 x64, RTX 5090 (sm_120a), and a CUDA 13.1-capable NVIDIA driver.

NInfer Windows 0.2.0 (portable, CUDA 13.1)

New in 0.2.0:

• Qwen3.8-27B NVFP4 artifact support (runtime + converter), validated end-to-end
• In-process Web UI serving (stock llama.cpp WebUI, same port as /v1)
• Serving fixes from upstream: unified multimodal context admission, ordered instruction turns, thinking-preserving prefix reuse

Self-contained portable zip: executables, all runtime DLLs, one launcher per supported model (all at 150k context), models\ folder, README.txt, SHA256SUMS. Verify against SHA256SUMS after download. Windows 11 x64, RTX 5090, CUDA 13.1-capable driver.

NInfer Windows 0.1.0 (portable, CUDA 13.1)

First portable Windows release of ninfer-windows — a fork of Neroued/ninfer with native Windows 11 x64 support.

This zip is self-contained: executables, all runtime DLLs (FFmpeg, libcurl, zlib, VC++ runtime; the CUDA runtime is statically linked), launcher scripts, a models\ folder, and SHA256SUMS for verification.

Requirements

• Windows 11 x64
• NVIDIA GeForce RTX 5090 with a driver supporting CUDA 13.1

Quick start

1. Extract the zip anywhere.
2. Download a model into models\ (links in the README at the top of the archive, or in the repo README).
3. Run the matching launcher, e.g. qwen3_8_27b.bat. This starts ninfer-serve on http://127.0.0.1:8080 with the API at /v1 and the llama.cpp WebUI at the root URL. The first run downloads the WebUI automatically, so it needs an internet connection.

The launchers default to a 150,000-token context to leave VRAM headroom for the Windows desktop; 200,000 is safe when VRAM is completely free at startup.

Changelog

• Native Windows 11 x64 build and run (MSVC + vcpkg; static CUDA runtime).
• Stock llama.cpp WebUI: ninfer-serve serves the unmodified UI in-process (--webui auto-downloads, --webui-dir DIR serves a local copy), and the server accepts the stock UI's API dialect.
• Portable release packaging with launcher scripts.