Skip to content

v0.1.0 — Experimental Qwen3 offline inference

Choose a tag to compare

@RedHeartSecretMan RedHeartSecretMan released this 26 Jul 15:41

English

Summary

Status: historical experimental release · released 2026-07-26.

v0.1.0 introduced single-GPU Qwen3 offline inference in Rust/CUDA, with reference fixtures from Transformers and vLLM. It remains available for reproducing early behaviour and studying its known accuracy failures; use v0.2.0 for new work.

Changes

  • Introduce synchronous LLM::new / LLM::generate, per-prompt sampling and a thin offline CLI.
  • Implement continuous batching, paged KV cache, prefix caching and recompute preemption for single-GPU Qwen3 generation.
  • Provide greedy, top-k and top-p sampling with repetition, presence and frequency penalties.
  • Attach Transformers/vLLM comparison fixtures with model/runtime provenance and per-fixture checksums.

Install and upgrade

git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git

Use the v0.2.0 getting-started guide for a supported build and generation walkthrough.

Verification

  • Fixture coverage: 28 prompt groups × two reference implementations = 56 fixtures; eight groups with full logits, 20 with top-5 data.
  • Full-output token agreement: four of eight groups passed, four differed.
  • Absolute logit tolerance recorded as 66.46875; six regression cases contain skipped positions.
  • No validated release performance baseline is claimed for this version.

Limitations

The historical numerical failures and skipped comparisons remain unchanged; this version provides no production accuracy guarantee. It is a synchronous offline engine without serving/async APIs or supported multi-GPU/TP/NCCL inference.

Links

简体中文

摘要

状态:历史实验性版本 · 发布于 2026-07-26。

v0.1.0 首次引入 Rust/CUDA 单 GPU Qwen3 离线推理,并附 Transformers 与 vLLM 参考夹具;保留用于复现早期行为与研究已知精度失败,新工作请使用 v0.2.0。

变更

  • 引入同步 LLM::new / LLM::generate、逐 prompt 采样与轻量离线 CLI。
  • 实现单 GPU Qwen3 生成所需的 continuous batching、分页 KV 缓存、前缀缓存与抢占重算。
  • 提供 greedy、top-k、top-p 采样,以及重复、存在与频率惩罚。
  • 附带带模型/运行时来源与逐夹具校验和的 Transformers/vLLM 对照夹具。

安装与升级

git clone --branch v0.1.0 --depth 1 https://github.com/RedHeartSecretMan/vllm-oxide.git

受支持的构建与生成流程见 v0.2.0 的快速开始指南。

验证

  • 夹具覆盖:28 个提示组 × 两种参考实现 = 56 份夹具;8 组含完整 logits,20 组含 top-5 数据。
  • 完整输出 token 一致性:8 组中 4 组通过、4 组存在差异。
  • 绝对 logit 容差记录为 66.46875;6 个回归用例含被跳过的位置。
  • 本版不声明任何经过验证的发布性能基线。

边界

历史数值失败与被跳过的比较保持不变,本版不提供生产级精度保证;它是同步离线引擎,不含 serving/异步 API 与受支持的多卡/TP/NCCL 推理。

链接