Skip to content

v0.2.0 — Single-GPU Qwen3 offline inference

Latest

Choose a tag to compare

@RedHeartSecretMan RedHeartSecretMan released this 13 Oct 09:13

English

Summary

Status: current release · released 2026-10-13.

v0.2.0 fixes generation errors in batching, cache reuse, recompute preemption and repeated calls, and runs Qwen3-0.6B/BF16 offline inference on one GPU through the synchronous Rust/CUDA API.

Changes

  • Preserve request identity, input/output order, token history and per-request sampling policy across scheduling and repeated generation.
  • Complete continuous batching, waiting admission, chunked prefill, prefix caching and recompute preemption with consistent causal history and token/KV budgets.
  • Fix numerical and batch-layout errors with independent QKV/MLP projections, explicit FP32 GEMM reduction, reference-ordered RMSNorm and call-layout-v2.
  • Automate Rust 1.94/stable, Python CPU tooling and dependency checks; validate real CUDA behaviour, model accuracy and performance on a local GPU.

Install and upgrade

Breaking pre-1.0 API upgrade. Crate-root exports are now LLM, EngineOptions, Prompt, SamplingParams, RequestOutput and Source; replace RequestOutput.seq_id with request_id. See the v0.1 → v0.2 migration guide.

git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
    hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234

Requires Rust 1.94+, Linux (including WSL2), an SM 8.9+ NVIDIA GPU and the CUDA Toolkit.

Verification

  • Independent numerical acceptance: 747/747 capture items and 231/231 execution groups passed (104 numerical comparisons, 44 behaviour checks, seven operator checks).
  • CUDA API/CLI smoke: production build, real warmup, 100 repeated API calls, three mixed requests, 107 unique request IDs and Quick Start output Paris..
  • RTX 4080/WSL2, 64-token greedy generation, gpu_memory_utilization = 0.50: 29.7601 decode token/s for one request (TTFT 40.9622 ms) and 115.8207 token/s total across a four-request batch.

Limitations

Serving/async/streaming APIs, multi-GPU/TP/NCCL, other model families, quantization, LoRA, speculative decoding and CUDA Graphs are outside this release. These baselines do not establish throughput parity with vLLM on other hardware.

Links

简体中文

摘要

状态:当前版本 · 发布于 2026-10-13。

修复批处理、缓存复用、抢占重算与重复调用中的生成错误,并通过同步 Rust/CUDA API 在单张 GPU 上完成 Qwen3-0.6B/BF16 离线推理。

变更

  • 在调度与重复生成之间保留请求身份、输入输出顺序、token 历史与各自的采样策略。
  • 补齐 continuous batching、等待准入、chunked prefill、前缀缓存与抢占重算,因果历史与 token/KV 预算保持一致。
  • 用独立 QKV/MLP 投影、显式 FP32 GEMM 归约、参考顺序 RMSNorm 与 call-layout-v2 修正数值与批布局错误。
  • 自动化 Rust 1.94/stable、Python CPU 工具与依赖检查;在本地 GPU 上验证真实 CUDA 行为、模型精度与性能。

安装与升级

破坏性预 1.0 API 升级。 crate 根导出收敛为 LLM、EngineOptions、Prompt、SamplingParams、RequestOutput、Source;RequestOutput.seq_id 改为 request_id。迁移见 v0.1 → v0.2 迁移指南。

git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
    hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234

验证

  • 独立数值验收:747/747 捕获项与 231/231 执行组通过(104 项数值比较、44 项行为检查、7 项算子检查)。
  • CUDA API/CLI 冒烟:生产构建、真实预热、100 次重复 API 调用、三路混合请求、107 个请求 ID,Quick Start 输出 Paris.。
  • RTX 4080/WSL2、64 token 贪心生成、gpu_memory_utilization = 0.50:单请求 29.7601 token/s(TTFT 40.9622 ms),四请求合计 115.8207 token/s。

边界

Serving/异步/流式 API、多卡与 TP/NCCL、其他模型族、量化、LoRA、投机解码与 CUDA Graphs 不在本版范围;以上基线不构成其他硬件上与 vLLM 的性能对等。

链接