Repository navigation
English
Summary
Status: current release · released 2026-10-13.
v0.2.0 fixes generation errors in batching, cache reuse, recompute preemption and repeated calls, and runs Qwen3-0.6B/BF16 offline inference on one GPU through the synchronous Rust/CUDA API.
Changes
- Preserve request identity, input/output order, token history and per-request sampling policy across scheduling and repeated generation.
- Complete continuous batching, waiting admission, chunked prefill, prefix caching and recompute preemption with consistent causal history and token/KV budgets.
- Fix numerical and batch-layout errors with independent QKV/MLP projections, explicit FP32 GEMM reduction, reference-ordered RMSNorm and
call-layout-v2. - Automate Rust 1.94/stable, Python CPU tooling and dependency checks; validate real CUDA behaviour, model accuracy and performance on a local GPU.
Install and upgrade
Breaking pre-1.0 API upgrade. Crate-root exports are now LLM, EngineOptions, Prompt, SamplingParams, RequestOutput and Source; replace RequestOutput.seq_id with request_id. See the v0.1 → v0.2 migration guide.
git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234Requires Rust 1.94+, Linux (including WSL2), an SM 8.9+ NVIDIA GPU and the CUDA Toolkit.
Verification
- Independent numerical acceptance: 747/747 capture items and 231/231 execution groups passed (104 numerical comparisons, 44 behaviour checks, seven operator checks).
- CUDA API/CLI smoke: production build, real warmup, 100 repeated API calls, three mixed requests, 107 unique request IDs and Quick Start output
Paris.. - RTX 4080/WSL2, 64-token greedy generation,
gpu_memory_utilization = 0.50: 29.7601 decode token/s for one request (TTFT 40.9622 ms) and 115.8207 token/s total across a four-request batch.
Limitations
Serving/async/streaming APIs, multi-GPU/TP/NCCL, other model families, quantization, LoRA, speculative decoding and CUDA Graphs are outside this release. These baselines do not establish throughput parity with vLLM on other hardware.
Links
- Full release note: 中文 · English
- CHANGELOG · Release history · Validation summary
简体中文
摘要
状态:当前版本 · 发布于 2026-10-13。
修复批处理、缓存复用、抢占重算与重复调用中的生成错误,并通过同步 Rust/CUDA API 在单张 GPU 上完成 Qwen3-0.6B/BF16 离线推理。
变更
- 在调度与重复生成之间保留请求身份、输入输出顺序、token 历史与各自的采样策略。
- 补齐 continuous batching、等待准入、chunked prefill、前缀缓存与抢占重算,因果历史与 token/KV 预算保持一致。
- 用独立 QKV/MLP 投影、显式 FP32 GEMM 归约、参考顺序 RMSNorm 与
call-layout-v2修正数值与批布局错误。 - 自动化 Rust 1.94/stable、Python CPU 工具与依赖检查;在本地 GPU 上验证真实 CUDA 行为、模型精度与性能。
安装与升级
破坏性预 1.0 API 升级。 crate 根导出收敛为 LLM、EngineOptions、Prompt、SamplingParams、RequestOutput、Source;RequestOutput.seq_id 改为 request_id。迁移见 v0.1 → v0.2 迁移指南。
git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234验证
- 独立数值验收:747/747 捕获项与 231/231 执行组通过(104 项数值比较、44 项行为检查、7 项算子检查)。
- CUDA API/CLI 冒烟:生产构建、真实预热、100 次重复 API 调用、三路混合请求、107 个请求 ID,Quick Start 输出
Paris.。 - RTX 4080/WSL2、64 token 贪心生成、
gpu_memory_utilization = 0.50:单请求 29.7601 token/s(TTFT 40.9622 ms),四请求合计 115.8207 token/s。
边界
Serving/异步/流式 API、多卡与 TP/NCCL、其他模型族、量化、LoRA、投机解码与 CUDA Graphs 不在本版范围;以上基线不构成其他硬件上与 vLLM 的性能对等。