PegaInfer v0.1.1

Prebuilt Qwen3-only server for Linux x86_64. The archive contains CUDA 13.0 runtime libraries and supports NVIDIA compute capabilities 8.x through 12.x. NVIDIA driver 580 or newer, glibc 2.35 or newer, and OpenSSL 3 are required. Model weights are not included.

What's Changed

• fix(deepseek-v2-lite): widen EP2 HF correctness gate by @CAICAIIs in #381
• fix(build): probe CUDA toolkit layouts in a shared openinfer-build crate by @FeathBow in #343
• test(qwen3): add real PEFT LoRA accuracy gate by @FeathBow in #369
• test(qwen3): de-flake lora_golden_gate via own-distribution regret; add TP=2 pass by @FeathBow in #385
• fix(deepseek-v2-lite): precompute NCCL MoE route plan by @CAICAIIs in #383
• docs(readme): refresh Qwen3.5 benchmark against vLLM 0.23.0 by @CAICAIIs in #386
• docs(readme): add star history chart by @xiaguan in #387
• feat(scheduler): add engine-side request lifecycle logs (admitted/fin… by @lulululu-debug in #382
• fix(qwen35): align GDN chunk_state AOT pointers :16 to fix sm_90 crash by @FeathBow in #390
• fix(build): explicitly enable tracing log bridge by @xwxb in #391
• fix(qwen35): batch step-tail logits and sampling by @CAICAIIs in #392
• chore(vllm-support): remove unused load_tokenizer_async by @xiaguan in #394
• chore(vllm): bump git deps to latest main for long-prompt 400 fix by @xiaguan in #402
• refactor(server): own per-model engine launch in model crates by @xiaguan in #404
• refactor(scheduler): collapse per-request output fan-out into one channel + demux by @xiaguan in #405
• fix(ci): restore green CI — sim fixture vocab_size + rustfmt by @xiaguan in #406
• test(vllm-frontend): lift bridge demux tests to sibling file, cull low-value cases by @xiaguan in #407
• fix(qwen35): add __syncthreads() to close the warp_norms WAR race in GDN decode by @FeathBow in #411
• fix(qwen): batch mixed FlashInfer sampling by @CAICAIIs in #413
• refactor(sample): converge per-model token selection + logprobs into openinfer-sample by @xiaguan in #417
• feat(deepseek-v2-lite): start/load logging by @Kokoro2336 in #415
• perf(qwen35): fuse gate/up MLP and tune decode GEMMs by @CAICAIIs in #421
• feat(qwen3): add green context SM partitioning by @xiaguan in #388
• fix(vllm-frontend): reject logprobs=-1 and cache_salt with 400 by @Rushit in #426
• fix(deepseek-v2-lite): add NCCL decode graph probe by @CAICAIIs in #429
• feat(qwen3): add profile-based KV cache sizing by @wdlin233 in #430
• fix(qwen3-scheduler): route echo+logprobs requests to prefill, not unified by @Rushit in #427
• chore(qwen3-scheduler): prune redundant fake-loop tests, extract test module by @xiaguan in #433
• bench(qwen): add mixed sampling HTTP benchmark gate by @CAICAIIs in #432
• fix(qwen3): opt-in GEMM-N reduction-order pin by @FeathBow in #428
• perf(qwen3): select decode attention path by batch, drop ctx>=1024 gate by @xiaguan in #437
• docs(skills): add vllm-bench multi-turn serving benchmark skill by @xiaguan in #424
• docs(skills): scrub local paths from vllm-bench skill by @xiaguan in #440
• feat(qwen3): DFlash speculative decoding by @xiaguan in #436
• fix(qwen3-dflash): post-merge correctness fixes — capture-shape gate, unified readiness, stream by @xiaguan in #442
• feat(deepseek-v2-lite): add mixed-request serving gate with per-request KV ownership by @CAICAIIs in #441
• docs(readme): add Slack community invite badge by @xiaguan in #444
• fix(qwen3): opt-in split-KV chunk-count reduction-order pin by @FeathBow in #438
• feat(dynamo): serve openinfer Qwen3 as a Dynamo backend worker, with a pure-Rust frontend and KV-aware routing by @xiaguan in #445
• chore(deps): bump vllm rust frontend to dedicated-runtime perf rev by @xiaguan in #447
• fix(frontend): drop HTTP-layer sampling guard to restore request-runtime isolation by @xiaguan in #448
• DeepSeek-V2-Lite: HTTP concurrency evidence, subgroup decode batching, and long-prompt fix by @CAICAIIs in #451
• feat(qwen35): scheduler-level chunked prefill (#375) by @scatyf3 in #431
• perf(qwen3): warp-per-token QK RMSNorm+RoPE prefill kernel by @xiaguan in #455
• fix(qwen3-dflash): set enable_kv_events in DFlash test launch options by @FeathBow in #457
• perf(qwen3): decouple batch-invariant split-KV size from Tuned cap by @FeathBow in #449
• fix(qwen3): assert NumericPolicy is immutable post-construction by @FeathBow in #458
• test(qwen35): cover chunked prefill resumed paged kv by @Mrtroll486 in #460
• fix(qwen3): fail loud under --batch-invariant instead of silent per-token fallback by @FeathBow in #462
• bench(deepseek-v2-lite): retain issue 279 benchmark snapshot by @CAICAIIs in #463
• perf(qwen3): add DSpark Markov drafting by @xiaguan in #468
• Lower Qwen3 speculative accept trace to debug by @xiaguan in #472
• feat(kernels): gate MoE third-party substrates by @xiaguan in #474
• Add GLM5.2 DP1/EP8 load-weight slice by @xiaguan in #476
• fix(vllm-frontend): report real KV capacity for engines that publish it by @FeathBow in #475
• feat(glm52): MLA decode brick — projection, absorb, cache-pack, FlashMLA sparse decode by @xiaguan in #477
• fix(glm52): gate kernels behind explicit feature by @xiaguan in #478
• chore(kv-cache): drop local lora_salt_hash, use upstream compute_salt_hash by @xiaguan in #480
• chore(qwen3): single-source split-KV config and pin warmup, measure report GEMMs under the production policy by @FeathBow in #473
• fix(qwen3): size decode page indices for prefix-shared views by @n-WN in #482
• chore(kernels): bump FlashInfer v0.6.12 -> v0.6.13 (#479) by @n-WN in #486
• perf(kernels): vectorize batched embedding gather to 16-byte row copies (#484) by @n-WN in #488
• fix(qwen3): make stream override an RAII guard by @FeathBow in #487
• feat(qwen3-kernel-report): full-forward manifest coverage + roofline rank pass (#456) by @n-WN in #485
• docs(roadmap): add 2026-H2 roadmap by @xiaguan in #493
• fix(kernels): format embedding batch test to satisfy cargo fmt CI by @xiaguan in #494
• refactor(qwen3): rename crate openinfer-qwen3-4b -> openinfer-qwen3 by @xiaguan in #495
• feat(glm52): DSA indexer kernel ops — cache, topk, slots, hadamard, DeepGEMM MQA logits by @xiaguan in #489
• fix(qwen3): disable cuda graph under tensor parallelism by @FeathBow in #496
• feat(bench): one-shot serving benchmark script by @xiaguan in #497
• feat(glm52): accuracy oracle harness — pinned HF glm_moe_dsa reference + hardcoded probe gate by @xiaguan in https://github.com/p...

OpenInfer 0.1.0

OpenInfer 0.1.0

OpenInfer 0.1.0 is the first source release of OpenInfer: a pure Rust + CUDA LLM inference engine with an OpenAI-compatible /v1/completions API.

This release marks the point where the Qwen3-4B/8B line is usable as a production-oriented small-model serving stack, while the large MoE model lines remain active work in progress.

What Is Included

• OpenAI-compatible completions serving through the vLLM Rust frontend.
• A Rust-native engine contract built around EngineHandle, request submission, and streamed TokenEvents.
• Per-model engines instead of one universal model abstraction.
• CUDA Graph decode path for stable steady-state serving.
• Continuous batching scheduler for Qwen3.
• Prefix cache backed by Dynamo KV block-management logic.
• Pegaflow KV offload integration for HBM to host memory recovery paths.
• Feature-gated model support:
  • Qwen3-4B / Qwen3-8B
  • Qwen3.5-4B
  • DeepSeek-V4
  • DeepSeek-V2-Lite
  • Kimi-K2
• Kernel layer using a mix of handwritten CUDA, cuBLAS, FlashInfer, Triton AOT, TileLang, and CuTe DSL.

Qwen3 Status

The Qwen3-4B/8B path is the most mature line in this release.

It includes:

• BF16 weight loading
• full-attention prefill and decode
• paged KV cache
• prefix cache
• continuous batching
• CUDA Graph decode
• greedy and sampling support
• LoRA routing support
• integration and golden-logit accuracy gates

On a single RTX 5090, Qwen3-4B is broadly competitive with vLLM 0.22.1 under vllm bench serve using the same client, same request stream, and same seed.

At low load, OpenInfer and vLLM are close on TTFT and TPOT. At medium load, vLLM is still ahead on decode TPOT. Under overload, OpenInfer reaches comparable or slightly higher saturated output throughput in the current sweep, but both engines are well past the healthy latency regime at that point.

Prefix Cache And KV Offload

OpenInfer is designed around a readable KV data plane.

For GPU prefix-cache hits, warm TTFT stays low as context grows. In the Qwen3-4B RTX 5090 sweep, OpenInfer stays ahead of vLLM on warm TTFT across tested prompt lengths up to 16k tokens.

Pegaflow integration provides a host-memory KV recovery path. In CPU-memory warm-hit mode, long prompts avoid full prefill and recover KV blocks from host memory, giving larger speedups as prompt length grows.

Why Rust

The goal of OpenInfer is not to rebuild every piece of the inference stack from scratch.

Instead, the project uses Rust to assemble components with explicit ownership boundaries:

• vLLM Rust frontend for protocol, tokenizer, streaming, and OpenAI compatibility.
• Dynamo KV block-management logic for prefix matching and KV lifecycle.
• Pegaflow for KV offload.
• Per-model engines for model-specific scheduling, state layout, and kernel DAGs.
• A kernel crate that keeps CUDA/Triton/TileLang/CuTe DSL assets feature-gated by model.

This keeps the serving path inspectable, avoids a large Python runtime in the hot service, and makes it easier to co-design routing, KV cache, scheduler, and kernels.

Known Limitations

• Qwen3 is the mature path in this release.
• DeepSeek-V4 and Kimi-K2 support exist, but serving performance and production polish are still catching up to vLLM.
• Some model lines require feature-specific build dependencies, such as Triton, TileLang, CuTe DSL, NCCL, or RDMA-related libraries.
• This is a source release, not a prebuilt binary distribution.
• No formal release workflow exists yet; this tag is intended as the first public 0.1.0 checkpoint.

Build

Default build targets Qwen3-4B:

cargo run --release -- --model-path /path/to/Qwen3-4B

Other model lines are feature-gated:

cargo run --release --features qwen35-4b -- --model-path /path/to/Qwen3.5-4B
cargo run --release --features deepseek-v4 -- --model-path /path/to/DeepSeek-V4
cargo run --release --features kimi-k2 -- --model-path /path/to/Kimi-K2

Closing

OpenInfer 0.1.0 is both a working inference engine and a statement of direction: inference systems should be built from smaller, readable, replaceable Rust components.

The long-term goal is for pieces such as the frontend, KV cache, offload layer, scheduler, and kernel tooling to become composable building blocks that can be reused outside this repository.