Skip to content

Home

Jump to bottom
Micky Zhang edited this page Nov 1, 2026 · 4 revisions

NInfer-Qwen

NInfer-Qwen serves Qwen3.8-27B GSQ-RCO IQ3_XXS at its full 262,144-token context on a single GeForce RTX 5060 Ti 16 GB. It is a C++/CUDA engine forked from NInfer-all, which descends from NInfer, and it is tuned for this one model on this one card.

At a glance
Model Qwen3.8-27B, ISTA-DASLab's GSQ-RCO IQ3_XXS GGUF with its MTP head (10.4 GB)
Card RTX 5060 Ti 16 GB (sm_120a), Windows 11 + WSL2
Context 262,144 tokens with one to three requests; up to 32 requests with smaller contexts
Speed (one request) 3,646 tok/s prefill and 88.7 tok/s decode on a 30K-token prompt; 2,143 and 66.0 at 160K
APIs OpenAI Chat Completions and Responses, Anthropic Messages, llama.cpp; WebUI and monitor
Features MTP speculative decoding, n-gram copy drafts, 4-bit KV cache, pinned-RAM conversation cache, tool calls, structured output (JSON schema, regex, GBNF), reasoning-loop guard

Pages

  • Installation: the prebuilt release, or WSL2, CUDA and the build.
  • Model preparation: sources, licenses and the conversion.
  • Serving: launch options, concurrency, RAM cache, the Windows launcher and API examples.
  • Performance: measurements, method, quality and the comparison with NInfer-all master.
  • Evaluation: the capability evaluation protocol and results (IFBench and AIME 2025 so far).
  • Changes from NInfer-all: what this line adds and removes.
  • Troubleshooting
  • Roadmap: open work and the parked optimisation queue.

Quick start

The latest release has prebuilt archives for the RTX 50 series; nothing to build. On Windows, download ninfer-5060-qwen-0.1.0-windows-wsl-x64.zip, extract it, double-click Install.cmd, then start Qwen3.8-27B from the Start menu (Installation). On Linux (Ubuntu 24.04 or newer):

curl -LO https://github.com/2-com-org/ninfer-5060-qwen/releases/download/v0.1.0/ninfer-5060-qwen-0.1.0-linux-x64.tar.gz
tar -xzf ninfer-5060-qwen-0.1.0-linux-x64.tar.gz
cd ninfer-5060-qwen-0.1.0-linux-x64
./download-model.sh
./serve.sh

Once it is running, open http://localhost:8080 for the WebUI or use the API at http://localhost:8080/v1.

Build from source

git clone https://github.com/2-com-org/ninfer-5060-qwen.git ~/ninfer-5060-qwen
~/ninfer-5060-qwen/scripts/qwen38/setup.sh
~/ninfer-5060-qwen/scripts/qwen38/serve.sh

setup.sh installs the build packages, builds the server (30-60 minutes) and downloads the converted model; Installation covers the CUDA Toolkit it needs and each step.

NInfer-Qwen

Clone this wiki locally