v0.2.2

CUDA Graph reuse now separates packet shape from request ownership across Qwen, Flash Next and ASR. Ordinary decode, mixed prefill and MTP share a common operand-table contract and native binding code. Replacing request owners updates addresses without capturing another graph of the same shape.

All supported model execution packages are rebuilt for 0.2.2. Model-specific GDN, attention, expert and audio arithmetic is retained. The release also includes native execution packages for Nemotron-3.5-Embed-1B and Qwen3-Reranker-0.6B, using their complete-sequence graph paths.

Assets include the Linux aarch64 engine, SM87 execution packages, matching model metadata, source and license notices, and SHA256SUMS. Existing quantized weight payloads can be reused with the matching metadata and execution package.

v0.2.1

Orinfer v0.2.1

Native Rust inference for Jetson AGX Orin 64GB (aarch64, CUDA SM87).

Changes

• Batch ordinary stochastic decode sampling across up to 128 requests, using the same FP64 sampling rules, independent RNG histories, penalties, grammar masks and per-request output semantics.
• Allocate large shared GPU workspaces on first use, with stable virtual addresses for CUDA Graph replay and admission budgets for pending text, image and audio work.
• Parallelize the sampler maximum/invalid reduction. On the measured eight-request Flash workload, sampler GPU time fell from 11.90 to 8.43 ms. Full naturally completed Flash requests improved 1.88% for short translations and 2.66% for longer translations versus the initial batched implementation; short-request throughput matched the preceding scalar baseline. Index-Translate-2B improved 3.39% at concurrency 1 and 6.16% at concurrency 8. These are medians of three interleaved runs on the same Orin, with seed 20261002, temperature 0.8, top_p 0.9, MTP/prefix cache off; request timings include prefill and first-use preparation.
• Add Qwen3-ASR-1.7B transcription, native audio preprocessing, packed-window encoding and request-private audio state.
• Share Qwen packet boundaries and integer kernel components, reuse W8 scales, preserve measured row selections, and add opt-in single-stream MTP depth exploration.
• Require Python 3.12 for offline tools. Pin our source-built Jetson Torch/torchvision releases and validate the compiler environment.

Downloads and compatibility

The engine, seven execution packages, and model metadata use .tar.xz; verify downloads with SHA256SUMS. The engine includes the complete matching source, build tools, LGPL licenses and third-party notices. Runtime deployment uses Rust and the CUDA driver; Python/Torch are offline build dependencies.

Execution packages cover Index-Translate-2B/9B/35B-A3B-preview, Qwen3.5-4B, Qwen3.8-27B, Qwen3.8-Flash-Next, and Qwen3-ASR-1.7B. Weights are distributed separately under their model licenses. An execution archive must match the execution_package digest in cache/model.json; installing a new archive alone does not change a model's pinned package. Existing schema-1 models continue to load, and batched sampling uses the new packages.

The accompanying model-metadata archive contains matching configuration and descriptors without weight payloads. Reuse the corresponding quantized model's GPU/CPU safetensors, install its execution archive with orinfer package install, then validate the assembled model before importing it. Follow the archive's README; original model directories can be retained for rollback.

Validation

• Local make check and offline reference checks passed.
• Sampling validation covered 45 exact mass/status comparisons across five vocabulary sizes, non-aligned tails, invalid-value isolation, 1–128 rows, owner reordering and changed-input Graph replay; Compute Sanitizer reported zero errors.
• Real Flash, 2B, 27B and ASR checks compared complete sampled token histories and committed request states, including cancellation/reuse and grammar removal. Lazy-workspace validation covered prefix restoration, MTP transitions, images, multiple images and real audio.
• Release archives are independently installed and their native plans compared with the prepared models. Release API smoke tests exercise the packaged binary and the rebuilt model libraries.

Changes since v0.2.0

v0.2.0

Orinfer v0.2.0

Native LLM and vision-language inference for Jetson AGX Orin 64GB, with a Rust runtime and TileLang GPU kernels.

Highlights

• Six model execution packages: Qwen3.8-27B, Qwen3.8-Flash-Next, Qwen3.5-4B, and Index-Translate-2B/9B/35B-A3B-preview.
• Shared dynamic-row execution, E8P Q4A8/Q2A8 mixed precision, INT8 KV, and FP32 GDN state.
• Adaptive native MTP, GPU sampling, shared GDN/attention components, and bounded parallel model loading.
• Lazy multi-model management with OpenAI Chat Completions and Ollama-compatible APIs.
• Resumable JSONL inference with continuous batching, throughput scheduling and prefix-aware planning.
• English, Chinese, Japanese and Korean READMEs with unified throughput and startup measurements.
• GDN request-parallel history updates follow the explicitly bound operation contract.

Downloads

The engine archive includes the CLI, four READMEs, guides, source.tar.xz, licenses and third-party notices. Execution filenames use the architecture specification, compute policy, GPU target and release version; full package digests are recorded in RELEASE.json. Each execution archive contains the model-specific Rust library, cubins and corresponding binding source. All archives use XZ compression; SHA256SUMS verifies every download.

Model	Execution package	Model data
Index-Translate-2B	qwen35-2b · A8	Hugging Face · ModelScope
Qwen3.5-4B	qwen35-4b · A8	Hugging Face · ModelScope
Index-Translate-9B	qwen35-9b · A8	Hugging Face · ModelScope
Qwen3.8-27B	qwen35-27b · A8	Hugging Face · ModelScope
Index-Translate-35B-A3B-preview	qwen35-35b-a3b · A8	Publishing separately
Qwen3.8-Flash-Next	flash-next · A8	Hugging Face · ModelScope

35B weight publication is running in the background on Hugging Face and ModelScope.

Download prepared model data separately and install the execution archive whose digest matches cache/model.json. Per-model cards provide complete v0.2.0 deployment commands. Model data remains under its upstream license; Orinfer and its execution packages use LGPL-3.0-or-later.

Validation

Local make check and offline checks passed, as did GitHub CI on x86_64 and aarch64. All six libraries were rebuilt from the release source. The release CLI installed all six archives into a fresh cache and reproduced their validated plans. Real API checks covered 48 naturally completed requests, MTP, prefix reuse, four concurrent requests, images and multiple images.

Performance and quality reference records benchmark conditions, file-cache effects on startup, BF16/FP8 comparison scope and accepted numerical differences in the optimized 9B execution.

Changes since v0.1.4

v0.1.4

Orinfer 0.1.4 expands native 27B execution and consolidates Flash Next kernels on Jetson AGX Orin 64GB (aarch64, CUDA SM87).

Changes

• Native Qwen3.8-27B execution for the improved two-stage E8P Q4A8 weights, shared dynamic projections, compressed INT8 KV attention prefetch, resumable layer-boundary prefill and batched MTP.
• Flash Next shares dynamic row templates across ordinary decode, MTP verification and mixed prefill/decode. Dynamic causal-state and vision kernels replace per-length implementations; common projection components and offline autotuning reduce duplicated paths.
• Mixed batching, prefill cost estimation and layer-boundary scheduling improve interaction between long prompts and short requests. Prefix restore and per-request recurrent, convolution, KV and MTP state remain isolated.
• orinfer package install installs verified native execution packages entirely in Rust. Ready-made deployments require no Python, PyTorch, TileLang or Rust compiler.
• Engine, execution and bundled source archives now use .tar.xz. The installer also accepts previously published .tar.gz execution archives.
• Rust 1.99.1, updated Rust dependencies and MiniJinja 3 compatibility. Offline tooling uses uv-managed Python environments; compiler image orinfer-compiler:0.1.2 reduces its pinned Python dependency set from 196 to 53 packages while preserving the tested Jetson CUDA/PyTorch stack.
• Retired experimental tools, kernels and obsolete execution paths have been removed.

Assets and compatibility

• orinfer-v0.1.4-linux-aarch64.tar.xz: Rust CLI, documentation, offline tools, corresponding workspace source and license notices.
• orinfer-v0.1.4-qwen3_5-int8-quality-sm87-execution.tar.xz: rebuilt 27B native execution package.
• orinfer-v0.1.4-flash_next-int8-quality-sm87-execution.tar.xz: rebuilt Flash Next native execution package.
• SHA256SUMS: SHA256 checksums for all archives.

Both execution packages contain native libraries rebuilt from the 0.1.4 workspace, generated binding source and rebuild instructions. Existing verified cubin payloads are retained. Weights are distributed separately and are not included in this release. Model metadata must reference the exact execution-package digest listed in the engine archive's RELEASE.json; a model pointing to an older digest does not automatically select a new package. Rebuilding prepared-model metadata/native libraries does not require requantizing weights. Only the current schema 1 / native ABI 1 format is supported.

See the bundled docs/performance.md for dated, workload-specific performance and quality results. This release does not claim that every context length or concurrency improves, and short release regressions do not replace an independent BF16 benchmark.

Validation

• Local make check: 221 Rust tests, 200 CPU Python tests, formatting, clippy and acceptance coverage checks passed. Both x86_64 and aarch64 GitHub CI jobs passed for the release commit.
• make check-offline: all 235 offline modules imported and 35 reference/contract tests passed in compiler image 0.1.2.
• Both native libraries were rebuilt with version 0.1.4. Complete expanded execution plans, bindings, ABI parameters and launches match the previously validated packages; immutable weight payloads are unchanged.
• The extracted release CLI installed both XZ execution archives through the Rust installer into an isolated shared cache. Both matching prepared models passed full validate-model checks.
• Actual release-binary inference passed for both models: 64 output tokens across eight fixed requests exactly match the previous binaries/packages at seed 20261002.
• Each model passed off, decode_only and full CUDA Graph release smoke tests, including streaming MTP generation and exact prefix replay. Additional decode_only requests covered default non-streaming JSON, bounded medium thinking, single/multiple images and two simultaneous requests. These are short functional regressions, not new long-generation performance benchmarks.

v0.1.3

Orinfer 0.1.3 improves Flash Next inference on Jetson AGX Orin 64GB and publishes rebuilt native execution packages for both supported architectures.

Highlights

• Flash Next exact E8P Q2/A8 expert kernels, Q4/W8 mixed projections, hyperconnection and convolution fusions, and cached CUDA Graph execution. Full expert residency remains the default; optional SSD expert paging uses the configurable residency budget.
• On the tested 512-token/128-output workload: about 1009 prefill tokens/s and 30.4 decode tokens/s without MTP. A complete Rust-code request with medium thinking, a 16K total output budget, MTP and prefix caching reached 45.19 decode tokens/s, producing 6,276 tokens before a natural stop. Measurements and timing scopes are documented in docs/performance.md.
• Compact schema 1 execution catalogs store shared compiled definitions instead of expanded layer bindings. Rust model libraries generate execution order and export bindings/programs through owned C ABI arrays. CUDA arithmetic and request-state contracts are preserved.
• Both Qwen3_5 27B and Flash Next execution packages are rebuilt with the 0.1.3 model library and include generated binding source and rebuild instructions.

Assets

• orinfer-v0.1.3-linux-aarch64.tar.gz: CLI, offline tools, documentation, workspace source and license notices.
• orinfer-v0.1.3-qwen3_5-int8-quality-sm87-execution.tar.gz: rebuilt 27B execution package.
• orinfer-v0.1.3-flash_next-int8-quality-sm87-execution.tar.gz: rebuilt optimized Flash Next execution package.
• SHA256SUMS: checksums for all archives.

Model weights are supplied separately. Install the matching execution package and use prepared model metadata referencing its digest. The online loader accepts the current schema 1 catalog structure; older packages containing expanded kernels lists are unsupported. Repacking existing bindings and rebuilding the native library does not require requantizing weight payloads. Execution packages include lib/bindings.rs and lib/BUILD.txt; the CLI archive supplies the matching workspace source.

Validation

Local make check and x86_64/aarch64 CI passed. Both rebuilt packages preserve the complete previous execution manifest, including ABI argument order, launches, buffer offsets and state contracts. Flash Next GPU validation covered all Graph modes, exact ordinary/MTP state and token comparisons, prefix restoration, MTP depths 1–7, dynamic concurrent requests, cancellation and slot reuse, and complete SSE code generation. The 0.1.3 CLI and rebuilt Flash Next package also pass release GPU/API smoke tests.

27B validation covers the rebuilt execution plan and native ABI; its deleted weight payloads prevent a new full 27B GPU performance run. This release does not claim a new full-model BF16 benchmark or a guaranteed startup speedup.

Flash Next SM87 execution package · 5bcf64efe5d8

Independent Flash Next int8_quality execution package for Jetson Orin SM87. Includes the native model library and a deduplicated safetensors bundle of cubin/source/host assets; model weights are distributed separately. Package digest: 5bcf64efe5d81703a02fba41753101eaaaaf6d39c1500580de32c1d747f2b1d8.

Use Orinfer built from commit 19a2320 or a newer compatible revision. Install the archive with:

PYTHONPATH=. .venv/bin/python tools/model/package.py install \
  ./flash_next-int8_quality-sm87-5bcf64efe5d81703a02fba41753101eaaaaf6d39c1500580de32c1d747f2b1d8.tar.gz ~/.cache/orinfer/packages

Validated against the Huihui Flash Next E8P Q2A8 model: complete tensor/CPU-table/kernel hash verification, exact execution-plan equivalence, fixed-seed output and top-3 probability agreement, multi-image input, prefix caching, streaming, concurrency, and MTP.

v0.1.2

Orinfer 0.1.2 brings native execution packages and Flash Next support to Jetson AGX Orin 64GB (aarch64, SM87).

Highlights

• Native model execution through C ABI v1, with a Rust shared library and matching TileLang cubins in each execution package.
• Flash Next Q2A8 with quality-oriented mixed precision, INT8 KV, images and multiple images, MTP, prefix caching and continuous batching.
• OpenAI Chat Completions streaming, incremental function calls, strict tools, JSON/schema constraints, thinking budgets, target logprobs and token bias; configuration examples for OpenCode and Pi.
• Packaging verifies the native library's actual version and capabilities and includes offline tools, source and license notices.
• Memory admission failures return HTTP 503 with service_unavailable and preserve the serving worker.
• CI runs on x86_64 and aarch64, including native ARM execution-package ABI tests.

Assets and compatibility

• orinfer-v0.1.2-linux-aarch64.tar.gz: CLI, tools, documentation, source and license notices.
• orinfer-v0.1.2-qwen3_5-int8-quality-sm87-execution.tar.gz: Qwen3_5 execution package.
• orinfer-v0.1.2-flash_next-int8-quality-sm87-execution.tar.gz: Flash Next execution package.
• SHA256SUMS: checksums for the three archives.

Model weights are supplied separately. Prepared models use the current schema 1 native execution-package structure; artifacts referencing older non-native operator packages must be rebuilt. Execution packages are matched by digest.

Validation

130 Rust tests, 141 CPU Python tests and 18 offline tests passed, together with x86_64/aarch64 CI. Flash Next GPU/API checks covered all three CUDA Graph modes, MTP, images/multiple images, four concurrent requests, prefix restoration, tools, constraints, logprobs and recovery after a capacity rejection. Qwen3_5 27B validation covered execution-plan generation and native ABI; this release does not include a new full 27B GPU run or a full-model BF16 quality benchmark.

v0.1.1

v0.1.1 汇总运行时加固、代码简化与依赖更新。支持 Jetson AGX Orin 64GB / aarch64 / CUDA SM87，继续提供 Qwen3.8-27B（Qwen3_5 架构）的文本、图片、多图和 OpenAI Chat Completions 服务。

更新

• 完善请求取消、服务关闭和 CUDA 故障处理，保护 VMM 分配回滚及请求私有状态，避免失败后复用损坏资源。
• 增加 CPU 预处理并发/内存预算、输出超时和关闭排空控制；加载时校验 tokenizer、chat template 与 generation config 的身份。
• 简化调度和生命周期代码，统一离线模型/算子包的原子发布与完整性校验。
• 使用 uv 管理 CPU .venv 和离线编译环境；更新 Rust 1.99.0、TileLang 0.1.15、Transformers 5.18.0、Safetensors 0.8.0 及 Rust 运行时依赖，适配序列化和视觉参考接口。
• 保留已验证的 Jetson CUDA 12.6 / PyTorch 2.9.1 组合。CI 使用固定版本的工具与固定 commit 的 Actions。

下载与部署

• orin-llm-v0.1.1-linux-aarch64.tar.gz：Rust 程序、源码、离线打包/安装工具及许可证。在线服务无需 Python；要求 aarch64 Linux、glibc >= 2.34 和 NVIDIA CUDA Driver。
• orin-llm-v0.1.1-qwen3_5-27b-int8-quality-sm87-operators.tar.gz：已验证算子包。复用 v0.1.0 的 AOT 产物，package digest 为 4c1ab6dd83f00d3e630f0bbe5595722acc8ebb5c9b40000c468f847e75e9df80；模型目录须引用匹配的包身份与物理布局。
• SHA256SUMS：下载后执行 sha256sum -c SHA256SUMS。

CPU 开发环境执行 make python-env 后使用 make check。离线模型准备和算子安装见模型构建说明，编译镜像默认名为 orin-llm-compiler:0.1.1。

本发布不附带权重；模型须准备为当前引擎原生目录，不直接导入任意 HF/AWQ/GGUF 权重。默认监听 0.0.0.0:8088，CUDA Graph 为 decode_only，支持 full/off。256K 上下文、prefix cache、连续批处理和 MTP 保持支持。

验证

• 85 项 Rust、64 项 CPU Python 和 2 项离线 MTP 导入测试通过。
• 三种 CUDA Graph 模式均通过真实模型的批处理、请求隔离、取消/重排/复用及文本/图片/多图质量回归；质量指标与依赖更新前相同。
• 新编译环境通过 SM87 norm、GDN 状态及视觉 kernel 的数值、尾部和改变输入的 graph replay 检查；12 个 INT8 投影用例与旧编译器输出逐字节一致。
• API/SSE、prefix 复用、固定 seed 采样、工具调用与服务关闭检查通过。

完整 LLM benchmark、独立 BF16 256K 评分、视频、其他架构和外部权重格式导入仍属于后续范围。tokenizers 间接依赖的 paste 有上游停止维护告警，暂无修复版本。

许可证：LGPL-3.0-or-later；第三方依赖和模型权重各遵循其许可证。

v0.1.0 — 第一阶段发布

Orin LLM 的首个版本，发布已完成第一阶段验收的提交 6045e790c487ba5da46d8e56cb48f5bcd3f897f5。

功能

• Jetson AGX Orin 64GB / aarch64 / CUDA SM87；Rust 在线运行时与 TileLang kernel。
• Qwen3.8-27B（Qwen3_5 架构）文本、图片及多图；原生 chat template、MTP、OpenAI Chat Completions、SSE 和工具调用。
• 常驻单份 W4 权重，INT8 为主的质量优先混合精度；按需分配 INT8 KV，FP32 GDN 状态。
• 256K 模型上下文配置、完整状态 prefix cache、连续批处理、联合 prefill、实际行数的非标准 batch 回退。
• CUDA Graph 支持 decode_only（默认）、full、off；默认监听 0.0.0.0:8088。

下载与部署

• orin-llm-v0.1.0-linux-aarch64.tar.gz：运行程序、使用说明、离线算子安装工具及许可证。已验证 Ubuntu 22.04 / aarch64；要求 glibc >= 2.34 和 NVIDIA CUDA Driver，kernel 构建使用 CUDA 12.6。
• orin-llm-v0.1.0-qwen3_5-27b-int8-quality-sm87-operators.tar.gz：已验收算子包；需模型目录引用匹配的 package digest、配置与物理布局。
• SHA256SUMS：下载后执行 sha256sum -c SHA256SUMS。
• GitHub 自动提供该标签的源码归档；构建及模型准备见 模型构建说明。

权重单独准备为引擎原生模型目录，本发布未附带权重，也不直接加载任意 HF/AWQ/GGUF 目录。在线推理不需要 Python；离线安装算子包需要 Python 3 和 safetensors，命令见运行程序归档内 README。

算子包 digest：4c1ab6dd83f00d3e630f0bbe5595722acc8ebb5c9b40000c468f847e75e9df80。

验收与限制

• 通过约定的同源 BF16 快速质量评测，以及近 256K 在线 Chat/MTP、prefix 重放和多轮续写；超预算请求准入前返回 400。
• 三种 CUDA Graph 模式通过真实模型的状态隔离、取消/复用、固定 seed 重排和质量回归。
• 189 个混合任务全部通过，包含 128 客户端提交及排队、文字/图片/多图、不同到达时间、分支和小缓存驱逐；API/SSE、停止词、采样及工具调用通过。
• 本机默认服务加载到可用约 24.1 秒。无 MTP 单流固定块测试：512/2K/8K prefill 分别约 676/819/774 TPS，decode 约 10.9/10.8/10.2 TPS；这是引擎固定 token-ID 路径，不是 API 或并发吞吐承诺。
• 近满上下文可能因内存压力仅部分命中缓存；此次重放/续写命中 65536 tokens，剩余约 194K 重新计算。不保证近 256K 完整缓存命中。
• 完整 LLM benchmark、独立 BF16 256K 评分、视频、其他模型架构及外部权重格式导入留待后续。

许可证：LGPL-3.0-or-later；第三方依赖及模型权重各遵循其许可证。