Skip to content

Releases: star1872/ninfer-offload

Release list

v0.4.0: expert cache — faster Flash-Next decode

v0.4.0: expert cache — faster Flash-Next decode Latest
Latest

Choose a tag to compare

View all tags
@star1872 star1872 released this 04 Oct 15:26
v0.4.0
1fd5c39

NInfer-Offload v0.4.0 — expert cache: faster Flash-Next decode

English | 中文

New

  • Expert cache (--moe-offload): the --moe-gpu-experts slots now follow recent routing. Experts the router keeps choosing are copied to the GPU in the background (at most two per layer and call, without stalling decode), replacing the least used ones. --moe-static-experts keeps the startup placement fixed (the v0.3.0 behaviour).
  • Faster expert routing on the GPU (the router no longer runs in a single thread block): fixed slots are also about 7% faster than v0.3.0.
  • ninfer-run downloads Flash-Next from Hugging Face (luocha2050/Qwen3.8-Flash-Next-NInfer-Offload) when huggingface.co is reachable and from ModelScope (mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload) otherwise. Both hold the same files.

Speed (RTX 4060 Ti 16 GB, Xeon E5-2673 v3, DDR3 ~21 GB/s on two channels, 64 GPU experts per layer, --max-context 2048, ninfer-serve, greedy, decode tok/s, four test prompts):

configuration essay code edit translation explanation
v0.3.0, no MTP 14.37 12.61 13.33 13.80
v0.4.0, fixed slots 15.34 13.56 14.28 14.77
v0.4.0, expert cache 24.08 19.17 19.16 21.76
v0.4.0, expert cache + 1 MTP draft 25.53 22.57 21.66 25.42

Package check on the same machine (./ninfer-run <model> --serve, automatic plan: context 8192, 65 GPU experts, 1 MTP draft): a ~270-token Chinese essay decoded at 26.1 tok/s and a ~250-token English explanation at 22.2 tok/s. Prefill is unchanged (about 30 tok/s on this machine).

Notes

  • The expert cache keeps the host expert banks page-locked (about 69 GB for Flash-Next); RAM needs are otherwise unchanged (about 80 GB for Flash-Next).
  • Routing and mathematics are unchanged, but greedy output can differ between runs because which side (GPU or CPU) computes an expert depends on timing. Long-text NLL with the cache (0.744) is within the spread of fixed placements (0.745-0.749). Use --moe-static-experts where bitwise repeatability matters.
  • ninfer-run needs python3. The model files from v0.3.0 work unchanged.
  • Built from commit 1fd5c39 with CUDA 12.8 for RTX 40-series (sm_89). Tested only on the machine above.

中文

新功能

  • 专家缓存（--moe-offload）：--moe-gpu-experts 的显存槽位会跟随最近的路由。路由反复选中的专家在后台复制进 GPU（每层每次最多 2 个，不阻塞解码），替换最少用的专家。--moe-static-experts 保持启动时的固定放置（即 v0.3.0 的行为）。
  • GPU 上的专家路由更快（路由不再由单个线程块计算）：固定放置也比 v0.3.0 快约 7%。
  • ninfer-run 下载 Flash-Next 时，能连上 huggingface.co 就从 Hugging Face（luocha2050/Qwen3.8-Flash-Next-NInfer-Offload）下载，否则从 ModelScope（mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload），两边文件相同。

速度：见上表。专家缓存让解码比 v0.3.0 快 1.4～1.7 倍；再开 1 个 MTP 草稿，在 4 个测试提示词上达到 21.7～25.5 tok/s。用安装包在同一台机器上实测（./ninfer-run 模型 --serve，自动方案：上下文 8192、65 个 GPU 专家、1 个 MTP 草稿）：约 270 token 的中文短文解码 26.1 tok/s，约 250 token 的英文解释 22.2 tok/s。预填充速度不变（这台机器上约 30 tok/s）。

说明

  • 专家缓存需要把主机上的专家权重锁页（Flash-Next 约 69 GB）；内存需求与之前相同（Flash-Next 约 80 GB）。
  • 路由和数学不变，但哪个专家由 GPU 还是 CPU 计算取决于时序，所以贪心输出在多次运行之间可能不同。开启缓存时长文本 NLL（0.744）在不同固定放置的波动范围内（0.745～0.749）。需要逐位可复现时请加 --moe-static-experts。
  • ninfer-run 需要 python3。v0.3.0 的模型文件可直接使用。
  • 由提交 1fd5c39 用 CUDA 12.8 为 RTX 40 系（sm_89）编译。只在上面这台机器上测试过。
Assets 4
Loading

v0.3.0: MTP speculative decoding for Flash-Next

v0.3.0: MTP speculative decoding for Flash-Next

Choose a tag to compare

View all tags
@star1872 star1872 released this 02 Oct 03:11
v0.3.0
862e9b9

NInfer-Offload v0.3.0 — MTP speculative decoding for Qwen3.8-Flash-Next

English | 中文

New

  • MTP speculative decoding for Flash-Next (--spec mtp --draft-tokens N on ninfer and ninfer-serve), with the model's own MTP predictor; its routed experts run on the CPU like the main model's.
  • ninfer-run turns MTP on with one draft when the model file contains it (--no-mtp, --draft-tokens N) and includes the MTP weights in its memory plan.
  • Converter --reuse FILE: add components (such as MTP) to an existing conversion without converting everything again.

Speed (RTX 4060 Ti 16 GB, Xeon E5-2673 v3, DDR3 ~21 GB/s, 64 GPU experts per layer, ninfer-serve, greedy, decode tok/s):

prompt no MTP 1 draft 2 drafts 3 drafts
essay 14.29 14.74 (+3%) 13.61 (-5%) 11.26 (-21%)
code edit 12.61 15.16 (+20%) 16.65 (+32%) 17.55 (+39%)
translation 13.30 15.77 (+19%) 15.51 (+17%) 14.97 (+13%)
explanation 13.69 16.65 (+22%) 17.49 (+28%) 16.87 (+23%)

One draft was faster on every prompt; more drafts help code and slow down free-form prose. Prefill is about 1-2% slower with MTP. MTP adds about 0.1 GiB of VRAM and 1.4 GB of RAM.

Notes

  • MTP needs a model file with the MTP component. The ModelScope conversion mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload is being updated to include it (110 GB); older binaries keep working with the new file (without MTP).
  • Greedy output with MTP is not bit-identical to greedy output without it (verifying several tokens at once changes rounding); checked against an FP32 reference, MTP output agrees with the reference as often as non-MTP output.
  • Tested only on the machine above.

中文

新功能

  • Flash-Next 支持 MTP 投机解码（ninfer 和 ninfer-serve 加 --spec mtp --draft-tokens N），使用模型自带的 MTP 预测层，它的路由专家和主模型一样在 CPU 上计算。
  • ninfer-run：模型文件带 MTP 时自动开启 1 个草稿（--no-mtp 关闭，--draft-tokens N 调整），内存规划包含 MTP 权重。
  • 转换器 --reuse FILE：给已有转换补上 MTP 等组件，不用全部重新转换。

速度：见上表。每轮 1 个草稿时所有提示词都更快（+3%～+22%）；2～3 个草稿对代码更快，但写散文会变慢。开启 MTP 后预填充约慢 1%～2%，额外占用约 0.1 GiB 显存和 1.4 GB 内存。

说明

  • MTP 需要包含 MTP 组件的模型文件。ModelScope 上的转换正在更新为含 MTP 的版本（110 GB）；旧版程序也能读取新文件（只是不开 MTP）。
  • 开启 MTP 后的贪心输出与不开时不完全一致（一次验证多个 token 会改变舍入）；与 FP32 参考实现对比，两者和参考实现的一致程度相当。
  • 只在上面这台机器上测试过。
Loading

v0.2.1: one command from a model name to a chat (ninfer-run)

v0.2.1: one command from a model name to a chat (ninfer-run)

Choose a tag to compare

View all tags
@star1872 star1872 released this 01 Oct 11:38
v0.2.1
bfebce3

New: one command from a model name to a chat

tar --zstd -xf ninfer-offload-v0.2.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.1-linux-x86_64-sm89
./ninfer-run Qwen/Qwen3.8-Flash-Next          # download the prebuilt model, then chat
./ninfer-run Qwen/Qwen3.6-35B-A3B --serve     # or keep an OpenAI/Anthropic API on port 8080

ninfer-run (needs python3, no PyTorch) downloads a ready .ninfer conversion into ~/ninfer-models (Flash-Next 108 GB from ModelScope; Qwen3.6-35B-A3B, Qwen3.6-27B and Qwen3.8-27B from Hugging Face, falling back to hf-mirror.com), reads the model's weight sizes, chooses --moe-offload, GPU experts, threads and context from free GPU and host memory, starts ninfer-serve and opens a terminal chat (/think, /reset, /exit). --prompt TEXT answers once, --dry-run prints the plan, --models-dir changes the storage directory, and arguments after -- go to the engine. Models without a ready conversion need the source tree (python -m tools.run MODEL converts them by streaming with the official recipe).

On the development machine (RTX 4060 Ti 16 GB, 94 GB RAM) it plans 66 GPU experts per layer for Flash-Next at 8192 context; chats from the package and from the source tree completed with correct multi-turn answers at about 12 tok/s.

Fixed

  • Downloads and streaming conversion from hf-mirror.com failed with HTTP 403 (the mirror rejects the default Python agent).

Engine binaries are unchanged from v0.2.0 (same sources; rebuilt from commit bfebce3 with CUDA 12.8.90 and GCC 13.3). Requirements unchanged: RTX 40-series (sm_89), driver ≥ 570, glibc ≥ 2.38, AVX2.

新增：一条命令从模型名到对话

tar --zstd -xf ninfer-offload-v0.2.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.1-linux-x86_64-sm89
./ninfer-run Qwen/Qwen3.8-Flash-Next          # 下载预转换模型，然后直接对话
./ninfer-run Qwen/Qwen3.6-35B-A3B --serve     # 或者在 8080 端口提供 OpenAI/Anthropic 接口

ninfer-run（需要 python3，不需要 PyTorch）会把现成的 .ninfer 模型下载到 ~/ninfer-models（Flash-Next 108 GB 来自 ModelScope；Qwen3.6-35B-A3B、Qwen3.6-27B、Qwen3.8-27B 来自 Hugging Face，连不上时改用 hf-mirror.com），读取模型各部分权重大小，根据空闲显存和内存自动决定 --moe-offload、GPU 专家数、线程数和上下文长度，启动 ninfer-serve 后直接在终端对话（/think、/reset、/exit）。--prompt 文本 只回答一次，--dry-run 只打印方案，--models-dir 修改存放目录，-- 之后的参数原样传给引擎。没有现成转换的模型需要用源码目录（python -m tools.run 模型名 会按官方配方流式转换）。

在开发机（RTX 4060 Ti 16 GB，94 GB 内存）上，Flash-Next 在 8192 上下文下自动规划为每层 66 个 GPU 专家；从安装包和源码分别运行，对话（含多轮）均正常完成，约 12 tok/s。

修复

  • 从 hf-mirror.com 下载或流式转换会报 HTTP 403（镜像站拒绝 Python 默认 User-Agent）。

推理引擎与 v0.2.0 相同（源码未变，基于提交 bfebce3 重新编译）。运行要求不变。

Loading

v0.2.0: Flash-Next long context (QSA), offload in ninfer-serve

v0.2.0: Flash-Next long context (QSA), offload in ninfer-serve

Choose a tag to compare

View all tags
@star1872 star1872 released this 01 Oct 08:38
v0.2.0
4e54bfc

New

  • Long contexts for Qwen3.8-Flash-Next. Contexts above 2051 tokens now use the model's QSA token selection (each query attends to its 512 highest-scoring 4-token blocks plus the recent tail); previously --max-context was capped at 2051. Requires the default --kv-dtype bf16 and text-only input. Up to 2051 tokens attention is dense and unchanged.
  • ninfer-serve accepts --moe-offload (--moe-gpu-experts, --moe-threads, --moe-expert-stats, --moe-record-stats), so Flash-Next can run behind the OpenAI/Anthropic-compatible API. Recommended: --host-kv-mib 1024.
  • ninfer-perplexity --token-nll FILE writes per-token NLL.
  • Source tree: streaming conversion now downloads only the byte ranges the next jobs need and deletes them after use; the default cache is 20 GB (was effectively the whole checkpoint).

Measured (RTX 4060 Ti 16 GB, Xeon E5-2673 v3, ninfer-serve, 64 GPU experts, greedy):

  • QSA selections equal an independent CPU oracle exactly in the operator test; a passkey placed at the start of a 3,352-token prompt is answered correctly; 24 greedy tokens after a 3,300-token prompt match the FP32 reference.
  • 3,327-token prompt: 11.7 tok/s decode, 36 tok/s prefill. 1,777-token prompt: 11.7-11.9 tok/s with --max-context 4096 vs 11.8-12.0 with --max-context 2048 (~1.2% cost); short prompts are 0.9-2.0% slower with --max-context above 2051, so keep 2048 if you do not need longer contexts.
  • Streaming conversion of Qwen3.6-35B-A3B with a 20 GB cache: 69.3 GB downloaded, peak cache 8.3 GB, result identical to the published artifact.

Download ninfer-offload-v0.2.0-linux-x86_64-sm89.tar.zst, then:

tar --zstd -xf ninfer-offload-v0.2.0-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.0-linux-x86_64-sm89
./ninfer --help

Requirements unchanged (RTX 40-series sm_89, driver ≥ 570, glibc ≥ 2.38, AVX2; see README.md inside). Built from commit 4e54bfc with CUDA 12.8.90 and GCC 13.3 on Ubuntu 24.04. Tested only on the machine above. The prebuilt Flash-Next model on ModelScope works unchanged: https://modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload

新增

  • Qwen3.8-Flash-Next 支持长上下文。 超过 2051 个 token 时使用模型自带的 QSA token 选择（每个查询只看得分最高的 512 个 4-token 块和最近的尾部 token）；之前 --max-context 最多 2051。需要默认的 --kv-dtype bf16，且只支持纯文本输入；2051 以内仍是原来的稠密注意力。
  • ninfer-serve 支持 --moe-offload（以及 --moe-gpu-experts、--moe-threads、--moe-expert-stats、--moe-record-stats），Flash-Next 可以通过 OpenAI / Anthropic 兼容接口提供服务。建议加 --host-kv-mib 1024。
  • ninfer-perplexity --token-nll FILE 可输出逐 token 的 NLL。
  • 源码：流式转换现在只下载接下来要用的字节范围，用完即删；默认缓存 20 GB（以前实际上会缓存几乎整个原版模型）。

实测（RTX 4060 Ti 16 GB，Xeon E5-2673 v3，ninfer-serve，64 个 GPU 专家，贪心解码）：

  • 算子测试中 QSA 选块结果与独立的 CPU 参考完全一致；放在 3352 token 提示词开头的密码能被正确答出；3300 token 后贪心生成的 24 个 token 与 FP32 参考一致。
  • 3327 token 提示词：解码 11.7 tok/s，prefill 36 tok/s。1777 token 提示词：--max-context 4096 为 11.7～11.9 tok/s，--max-context 2048 为 11.8～12.0（约 1.2% 代价）；短提示词在上限超过 2051 时慢 0.9～2.0%，不需要长上下文时可继续用 2048。
  • Qwen3.6-35B-A3B 流式转换（缓存上限 20 GB）：下载 69.3 GB，缓存峰值 8.3 GB，结果与官方发布的模型一致。

下载 ninfer-offload-v0.2.0-linux-x86_64-sm89.tar.zst 后：

tar --zstd -xf ninfer-offload-v0.2.0-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.0-linux-x86_64-sm89
./ninfer --help

运行要求不变。只在上述机器上测试过。ModelScope 上的 Flash-Next 预转换模型无需更新即可使用。

Loading

v0.1.1 — multi-file model fix

v0.1.1 — multi-file model fix

Choose a tag to compare

View all tags
@star1872 star1872 released this 30 Sep 16:04
v0.1.1
b804a33

Bug-fix release. Upgrade from v0.1.0 if you run multi-file .ninfer models with --moe-offload.

Fixed

  • --moe-offload failed with a mapped object must lie in one container file when a model is split into several files (xxx.ninfer + xxx.ninfer.part-000N, the converter's default above 32 GB). Objects that cross files are now mapped file by file into one address range. Required for the prebuilt Flash-Next model on ModelScope: https://modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload
  • python -m tools.convert.wizard --help now prints help instead of starting the questions.

New (source tree): python -m tools.artifact.reshard in.ninfer out/in.ninfer --max-file-gb 50 rewrites an existing model into a different file split without re-converting.

Download ninfer-offload-v0.1.1-linux-x86_64-sm89.tar.zst, then:

tar --zstd -xf ninfer-offload-v0.1.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.1.1-linux-x86_64-sm89
./ninfer --help

Requirements and contents are unchanged from v0.1.0 (RTX 40-series sm_89, driver ≥ 570, glibc ≥ 2.38, AVX2; see README.md inside the archive).

Built from commit b804a33 with CUDA 12.8.90 and GCC 13.3 on Ubuntu 24.04. Tested only on RTX 4060 Ti 16 GB: Qwen3.8-Flash-Next from a 3-file model gives the same perplexity (mean NLL 1.443300) and the same greedy tokens as the single-file model; 13.1 tok/s decode with --moe-gpu-experts 64.

问题修复版本。使用多文件 .ninfer 模型并开启 --moe-offload 的用户请从 v0.1.0 升级。

修复

  • 模型分成多个文件时（xxx.ninfer + xxx.ninfer.part-000N，转换器超过 32 GB 默认如此），--moe-offload 会报错 a mapped object must lie in one container file。现在跨文件的对象会逐个文件映射到同一段连续地址。ModelScope 上的 Flash-Next 预转换模型（https://modelscope.cn/models/mymodel3861/Qwen3.8-Flash-Next-NInfer-Offload）需要此版本。
  • python -m tools.convert.wizard --help 现在显示帮助，而不是直接开始提问。

新增（源码）：python -m tools.artifact.reshard in.ninfer out/in.ninfer --max-file-gb 50 可以把已有模型重新分成不同大小的文件，无需重新转换。

下载 ninfer-offload-v0.1.1-linux-x86_64-sm89.tar.zst 后：

tar --zstd -xf ninfer-offload-v0.1.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.1.1-linux-x86_64-sm89
./ninfer --help

运行要求与内容同 v0.1.0。目前只在 RTX 4060 Ti 16 GB 上测试过：Qwen3.8-Flash-Next 三文件模型与单文件模型的困惑度（平均 NLL 1.443300）和贪心输出完全一致；--moe-gpu-experts 64 时生成速度 13.1 tok/s。

Loading

v0.1.0 — prebuilt Linux binaries for RTX 40-series

v0.1.0 — prebuilt Linux binaries for RTX 40-series

Choose a tag to compare

View all tags
@star1872 star1872 released this 30 Sep 08:07
v0.1.0
dfd60a7

First prebuilt release of NInfer-Offload: run 100B+ MoE models such as Qwen3.8-Flash-Next on an RTX 40-series GPU with the routed experts in system memory.

Download ninfer-offload-v0.1.0-linux-x86_64-sm89.tar.zst, then:

tar --zstd -xf ninfer-offload-v0.1.0-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.1.0-linux-x86_64-sm89
./ninfer --help

Requirements: NVIDIA RTX 40-series (sm_89), driver ≥ 570, Ubuntu 24.04 or another x86-64 Linux with glibc ≥ 2.38, AVX2 CPU, libavformat60 libavcodec60 libswscale7 libcurl4. The CUDA runtime is bundled. See README.md inside the archive.

Contents: ninfer (CLI), ninfer-perplexity, ninfer-serve (HTTP server; --moe-offload not yet supported there), routing statistics for Flash-Next and Qwen3.6-35B-A3B.

Model files: convert with python -m tools.convert.wizard — see the converter guide.

Built from commit dfd60a7 with CUDA 12.8.90 and GCC 13.3 on Ubuntu 24.04. Tested only on RTX 4060 Ti 16 GB.

NInfer-Offload 第一个预编译版本：在 RTX 40 系显卡上运行 Qwen3.8-Flash-Next 等 100B+ MoE 模型，路由专家放在内存中。

下载 ninfer-offload-v0.1.0-linux-x86_64-sm89.tar.zst 后：

tar --zstd -xf ninfer-offload-v0.1.0-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.1.0-linux-x86_64-sm89
./ninfer --help

要求：NVIDIA RTX 40 系（sm_89）、驱动 ≥ 570、Ubuntu 24.04 或 glibc ≥ 2.38 的 x86-64 Linux、支持 AVX2 的 CPU、libavformat60 libavcodec60 libswscale7 libcurl4。CUDA 运行库已包含。详见压缩包内 README.md。

模型文件：用 python -m tools.convert.wizard 转换，见转换器使用说明。

目前只在 RTX 4060 Ti 16 GB 上测试过。

Loading