Repository navigation
Releases
v0.2.0 — Single-GPU Qwen3 offline inference
English
Summary
Status: current release · released 2026-10-13.
v0.2.0 fixes generation errors in batching, cache reuse, recompute preemption and repeated calls, and runs Qwen3-0.6B/BF16 offline inference on one GPU through the synchronous Rust/CUDA API.
Changes
- Preserve request identity, input/output order, token history and per-request sampling policy across scheduling and repeated generation.
- Complete continuous batching, waiting admission, chunked prefill, prefix caching and recompute preemption with consistent causal history and token/KV budgets.
- Fix numerical and batch-layout errors with independent QKV/MLP projections, explicit FP32 GEMM reduction, reference-ordered RMSNorm and
call-layout-v2. - Automate Rust 1.94/stable, Python CPU tooling and dependency checks; validate real CUDA behaviour, model accuracy and performance on a local GPU.
Install and upgrade
Breaking pre-1.0 API upgrade. Crate-root exports are now LLM, EngineOptions, Prompt, SamplingParams, RequestOutput and Source; replace RequestOutput.seq_id with request_id. See the v0.1 → v0.2 migration guide.
git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234Requires Rust 1.94+, Linux (including WSL2), an SM 8.9+ NVIDIA GPU and the CUDA Toolkit.
Verification
- Independent numerical acceptance: 747/747 capture items and 231/231 execution groups passed (104 numerical comparisons, 44 behaviour checks, seven operator checks).
- CUDA API/CLI smoke: production build, real warmup, 100 repeated API calls, three mixed requests, 107 unique request IDs and Quick Start output
Paris.. - RTX 4080/WSL2, 64-token greedy generation,
gpu_memory_utilization = 0.50: 29.7601 decode token/s for one request (TTFT 40.9622 ms) and 115.8207 token/s total across a four-request batch.
Limitations
Serving/async/streaming APIs, multi-GPU/TP/NCCL, other model families, quantization, LoRA, speculative decoding and CUDA Graphs are outside this release. These baselines do not establish throughput parity with vLLM on other hardware.
Links
- Full release note: 中文 · English
- CHANGELOG · Release history · Validation summary
简体中文
摘要
状态:当前版本 · 发布于 2026-10-13。
修复批处理、缓存复用、抢占重算与重复调用中的生成错误,并通过同步 Rust/CUDA API 在单张 GPU 上完成 Qwen3-0.6B/BF16 离线推理。
变更
- 在调度与重复生成之间保留请求身份、输入输出顺序、token 历史与各自的采样策略。
- 补齐 continuous batching、等待准入、chunked prefill、前缀缓存与抢占重算,因果历史与 token/KV 预算保持一致。
- 用独立 QKV/MLP 投影、显式 FP32 GEMM 归约、参考顺序 RMSNorm 与
call-layout-v2修正数值与批布局错误。 - 自动化 Rust 1.94/stable、Python CPU 工具与依赖检查;在本地 GPU 上验证真实 CUDA 行为、模型精度与性能。
安装与升级
破坏性预 1.0 API 升级。 crate 根导出收敛为 LLM、EngineOptions、Prompt、SamplingParams、RequestOutput、Source;RequestOutput.seq_id 改为 request_id。迁移见 v0.1 → v0.2 迁移指南。
git clone --branch v0.2.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git
cd vllm-oxide
cargo run --locked --release -p vllm_oxide --example generate_qwen3 --features cuda -- \
hub:Qwen/Qwen3-0.6B@7e4ae267688d671ddfca3122e4528ee980cf3234验证
- 独立数值验收:747/747 捕获项与 231/231 执行组通过(104 项数值比较、44 项行为检查、7 项算子检查)。
- CUDA API/CLI 冒烟:生产构建、真实预热、100 次重复 API 调用、三路混合请求、107 个请求 ID,Quick Start 输出
Paris.。 - RTX 4080/WSL2、64 token 贪心生成、
gpu_memory_utilization = 0.50:单请求 29.7601 token/s(TTFT 40.9622 ms),四请求合计 115.8207 token/s。
边界
Serving/异步/流式 API、多卡与 TP/NCCL、其他模型族、量化、LoRA、投机解码与 CUDA Graphs 不在本版范围;以上基线不构成其他硬件上与 vLLM 的性能对等。
链接
v0.2.0-rc.1 — Qwen3 offline inference release candidate
English
Summary
Status: historical experimental prerelease · released 2026-09-10.
RC1 opened the Rust/CUDA single-GPU Qwen3 offline engine for testing and feedback while numerical and performance validation was still incomplete. For a new application, use v0.2.0.
Changes
- Expose Qwen3-0.6B/BF16 through the synchronous single-GPU
LLM::generateinterface. - Add continuous batching, paged KV cache, prefix caching, waiting admission, chunked prefill and recompute preemption.
- Introduce bounded supervision and validation tooling while retaining the original numerical budgets and measurement identities.
- Prepare the prerelease version and forward the CLI
cudafeature; RC finalization did not alter inference operators, weights or validation inputs.
Install and upgrade
The recorded environment is RTX 4080/WSL2 with Rust 1.94.0. For the supported API, upgrade to v0.2.0 and follow the migration guide. The Git tag v0.2.0-rc.1 matches the Cargo package version 0.2.0-rc.1.
Verification
- Capture completeness: 487/487 capture items and 153/153 capture groups verified.
- Numerical comparisons: 48 of 66 passed, 18 failed (all 15 development failures exceeded peak KL 0.005).
- Seven operator profiles and 22 behaviour checks passed; short generations and CPU CI passed.
- Full performance validation and the declared Rust minimum were incomplete.
Limitations
RC1 has no validated performance guarantee and its numerical failures remain unresolved. The later v0.2.0 acceptance applies to that release's own source and leaves this record unchanged.
Links
- Full release note: 中文 · English
- CHANGELOG · Release history
简体中文
摘要
状态:历史实验性预发布 · 发布于 2026-09-10。
RC1 把 Rust/CUDA 单 GPU Qwen3 离线引擎开放测试,当时数值与性能验证仍未完成;新项目请使用 v0.2.0。
变更
- 通过同步单 GPU
LLM::generate接口提供 Qwen3-0.6B/BF16。 - 增加 continuous batching、分页 KV 缓存、前缀缓存、等待准入、chunked prefill 与抢占重算。
- 引入有界监督与验证工具,同时保留原有数值预算与测量身份。
- 准备预发布版本并透出 CLI 的
cudafeature;RC 收尾未改动推理算子、权重或验证输入。
安装与升级
记录环境为 RTX 4080/WSL2 与 Rust 1.94.0。需要受支持的 API 与当前环境要求时,请升级到 v0.2.0 并参考迁移指南;Git tag v0.2.0-rc.1 与 Cargo 包版本 0.2.0-rc.1 一致。
验证
- 捕获完整性:487/487 捕获项与 153/153 捕获组通过校验。
- 数值比较:66 项中 48 通过、18 失败(15 项开发集失败均超过峰值 KL 0.005)。
- 7 项算子画像与 22 项行为检查通过;短生成与 CPU CI 通过。
- 完整性能验证与声明的 Rust 最低版本未完成。
边界
RC1 没有经过验证的性能保证,数值失败仍未解决;后续 v0.2.0 的验收只针对该版本自身源码,不改变本条记录。
链接
v0.1.0 — Experimental Qwen3 offline inference
English
Summary
Status: historical experimental release · released 2026-07-26.
v0.1.0 introduced single-GPU Qwen3 offline inference in Rust/CUDA, with reference fixtures from Transformers and vLLM. It remains available for reproducing early behaviour and studying its known accuracy failures; use v0.2.0 for new work.
Changes
- Introduce synchronous
LLM::new/LLM::generate, per-prompt sampling and a thin offline CLI. - Implement continuous batching, paged KV cache, prefix caching and recompute preemption for single-GPU Qwen3 generation.
- Provide greedy, top-k and top-p sampling with repetition, presence and frequency penalties.
- Attach Transformers/vLLM comparison fixtures with model/runtime provenance and per-fixture checksums.
Install and upgrade
git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.gitUse the v0.2.0 getting-started guide for a supported build and generation walkthrough.
Verification
- Fixture coverage: 28 prompt groups × two reference implementations = 56 fixtures; eight groups with full logits, 20 with top-5 data.
- Full-output token agreement: four of eight groups passed, four differed.
- Absolute logit tolerance recorded as
66.46875; six regression cases contain skipped positions. - No validated release performance baseline is claimed for this version.
Limitations
The historical numerical failures and skipped comparisons remain unchanged; this version provides no production accuracy guarantee. It is a synchronous offline engine without serving/async APIs or supported multi-GPU/TP/NCCL inference.
Links
- Full release note: 中文 · English
- CHANGELOG · Release history · Upgrade to v0.2.0
简体中文
摘要
状态:历史实验性版本 · 发布于 2026-07-26。
v0.1.0 首次引入 Rust/CUDA 单 GPU Qwen3 离线推理,并附 Transformers 与 vLLM 参考夹具;保留用于复现早期行为与研究已知精度失败,新工作请使用 v0.2.0。
变更
- 引入同步
LLM::new/LLM::generate、逐 prompt 采样与轻量离线 CLI。 - 实现单 GPU Qwen3 生成所需的 continuous batching、分页 KV 缓存、前缀缓存与抢占重算。
- 提供 greedy、top-k、top-p 采样,以及重复、存在与频率惩罚。
- 附带带模型/运行时来源与逐夹具校验和的 Transformers/vLLM 对照夹具。
安装与升级
git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git受支持的构建与生成流程见 v0.2.0 的快速开始指南。
验证
- 夹具覆盖:28 个提示组 × 两种参考实现 = 56 份夹具;8 组含完整 logits,20 组含 top-5 数据。
- 完整输出 token 一致性:8 组中 4 组通过、4 组存在差异。
- 绝对 logit 容差记录为
66.46875;6 个回归用例含被跳过的位置。 - 本版不声明任何经过验证的发布性能基线。
边界
历史数值失败与被跳过的比较保持不变,本版不提供生产级精度保证;它是同步离线引擎,不含 serving/异步 API 与受支持的多卡/TP/NCCL 推理。
链接
- 完整发布说明: 中文 · English
- 变更日志 · 发布历史 · 升级到 v0.2.0