Skip to content

v0.2.1: one command from a model name to a chat (ninfer-run)

Choose a tag to compare

View all tags
@star1872 star1872 released this 01 Oct 11:38
v0.2.1
bfebce3

New: one command from a model name to a chat

tar --zstd -xf ninfer-offload-v0.2.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.1-linux-x86_64-sm89
./ninfer-run Qwen/Qwen3.8-Flash-Next          # download the prebuilt model, then chat
./ninfer-run Qwen/Qwen3.6-35B-A3B --serve     # or keep an OpenAI/Anthropic API on port 8080

ninfer-run (needs python3, no PyTorch) downloads a ready .ninfer conversion into ~/ninfer-models (Flash-Next 108 GB from ModelScope; Qwen3.6-35B-A3B, Qwen3.6-27B and Qwen3.8-27B from Hugging Face, falling back to hf-mirror.com), reads the model's weight sizes, chooses --moe-offload, GPU experts, threads and context from free GPU and host memory, starts ninfer-serve and opens a terminal chat (/think, /reset, /exit). --prompt TEXT answers once, --dry-run prints the plan, --models-dir changes the storage directory, and arguments after -- go to the engine. Models without a ready conversion need the source tree (python -m tools.run MODEL converts them by streaming with the official recipe).

On the development machine (RTX 4060 Ti 16 GB, 94 GB RAM) it plans 66 GPU experts per layer for Flash-Next at 8192 context; chats from the package and from the source tree completed with correct multi-turn answers at about 12 tok/s.

Fixed

  • Downloads and streaming conversion from hf-mirror.com failed with HTTP 403 (the mirror rejects the default Python agent).

Engine binaries are unchanged from v0.2.0 (same sources; rebuilt from commit bfebce3 with CUDA 12.8.90 and GCC 13.3). Requirements unchanged: RTX 40-series (sm_89), driver ≥ 570, glibc ≥ 2.38, AVX2.

新增：一条命令从模型名到对话

tar --zstd -xf ninfer-offload-v0.2.1-linux-x86_64-sm89.tar.zst
cd ninfer-offload-v0.2.1-linux-x86_64-sm89
./ninfer-run Qwen/Qwen3.8-Flash-Next          # 下载预转换模型，然后直接对话
./ninfer-run Qwen/Qwen3.6-35B-A3B --serve     # 或者在 8080 端口提供 OpenAI/Anthropic 接口

ninfer-run（需要 python3，不需要 PyTorch）会把现成的 .ninfer 模型下载到 ~/ninfer-models（Flash-Next 108 GB 来自 ModelScope；Qwen3.6-35B-A3B、Qwen3.6-27B、Qwen3.8-27B 来自 Hugging Face，连不上时改用 hf-mirror.com），读取模型各部分权重大小，根据空闲显存和内存自动决定 --moe-offload、GPU 专家数、线程数和上下文长度，启动 ninfer-serve 后直接在终端对话（/think、/reset、/exit）。--prompt 文本 只回答一次，--dry-run 只打印方案，--models-dir 修改存放目录，-- 之后的参数原样传给引擎。没有现成转换的模型需要用源码目录（python -m tools.run 模型名 会按官方配方流式转换）。

在开发机（RTX 4060 Ti 16 GB，94 GB 内存）上，Flash-Next 在 8192 上下文下自动规划为每层 66 个 GPU 专家；从安装包和源码分别运行，对话（含多轮）均正常完成，约 12 tok/s。

修复

  • 从 hf-mirror.com 下载或流式转换会报 HTTP 403（镜像站拒绝 Python 默认 User-Agent）。

推理引擎与 v0.2.0 相同（源码未变，基于提交 bfebce3 重新编译）。运行要求不变。

Assets 4
Loading