Repository navigation
v0.1.0 — Experimental Qwen3 offline inference
English
Summary
Status: historical experimental release · released 2026-07-26.
v0.1.0 introduced single-GPU Qwen3 offline inference in Rust/CUDA, with reference fixtures from Transformers and vLLM. It remains available for reproducing early behaviour and studying its known accuracy failures; use v0.2.0 for new work.
Changes
- Introduce synchronous
LLM::new/LLM::generate, per-prompt sampling and a thin offline CLI. - Implement continuous batching, paged KV cache, prefix caching and recompute preemption for single-GPU Qwen3 generation.
- Provide greedy, top-k and top-p sampling with repetition, presence and frequency penalties.
- Attach Transformers/vLLM comparison fixtures with model/runtime provenance and per-fixture checksums.
Install and upgrade
git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.gitUse the v0.2.0 getting-started guide for a supported build and generation walkthrough.
Verification
- Fixture coverage: 28 prompt groups × two reference implementations = 56 fixtures; eight groups with full logits, 20 with top-5 data.
- Full-output token agreement: four of eight groups passed, four differed.
- Absolute logit tolerance recorded as
66.46875; six regression cases contain skipped positions. - No validated release performance baseline is claimed for this version.
Limitations
The historical numerical failures and skipped comparisons remain unchanged; this version provides no production accuracy guarantee. It is a synchronous offline engine without serving/async APIs or supported multi-GPU/TP/NCCL inference.
Links
- Full release note: 中文 · English
- CHANGELOG · Release history · Upgrade to v0.2.0
简体中文
摘要
状态:历史实验性版本 · 发布于 2026-07-26。
v0.1.0 首次引入 Rust/CUDA 单 GPU Qwen3 离线推理,并附 Transformers 与 vLLM 参考夹具;保留用于复现早期行为与研究已知精度失败,新工作请使用 v0.2.0。
变更
- 引入同步
LLM::new/LLM::generate、逐 prompt 采样与轻量离线 CLI。 - 实现单 GPU Qwen3 生成所需的 continuous batching、分页 KV 缓存、前缀缓存与抢占重算。
- 提供 greedy、top-k、top-p 采样,以及重复、存在与频率惩罚。
- 附带带模型/运行时来源与逐夹具校验和的 Transformers/vLLM 对照夹具。
安装与升级
git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git受支持的构建与生成流程见 v0.2.0 的快速开始指南。
验证
- 夹具覆盖:28 个提示组 × 两种参考实现 = 56 份夹具;8 组含完整 logits,20 组含 top-5 数据。
- 完整输出 token 一致性:8 组中 4 组通过、4 组存在差异。
- 绝对 logit 容差记录为
66.46875;6 个回归用例含被跳过的位置。 - 本版不声明任何经过验证的发布性能基线。
边界
历史数值失败与被跳过的比较保持不变,本版不提供生产级精度保证;它是同步离线引擎,不含 serving/异步 API 与受支持的多卡/TP/NCCL 推理。
链接
- 完整发布说明: 中文 · English
- 变更日志 · 发布历史 · 升级到 v0.2.0