Ferrite 0.4.3

Ferrite 0.4.3 tunes decoding for the RTX 5060 Ti 16 GB.

In this release

• Decode plans for the RTX 5060 Ti 16 GB. The EXL3 decode kernels have a plan table tuned on the card (36 SMs); until now it ran with the DGX Spark's.
• Faster generation on that card. Code goes from 139 to 155 tokens per second at 2K context (121 to 136 at 16K, 86 to 97 at 56K), and 500-token code edits from 343 to 357.
• Same text. Replies with drafting on and off are identical, as before.
• The DGX Spark and RTX 5080 engines are unchanged.

Details: docs/rtx-5080.md. Full list: CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.4.3-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The RTX 5080, RTX 5060 Ti and Arc Pro B70 targets are built from source.
• ferrite-0.4.3-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Ferrite 0.4.2

Ferrite 0.4.2 finishes the model id change of 0.4.1 in the model cards.

In this release

• Model cards use the model id qwen3.8-27b. The cards of both quants on Hugging Face, and their copies in docs/, showed the old ids spark and b70 in their serve examples.
• scripts/models.manifest pins the re-uploaded revisions. Only the card and the checksum list changed on Hugging Face; the weights are the same files.
• No engine change from 0.4.1.

Full list: CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.4.2-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The RTX 5080 and Arc Pro B70 targets are built from source.
• ferrite-0.4.2-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Ferrite 0.4.1

Ferrite 0.4.1 gives the model a name that makes sense on every machine and fixes the 16 GB preset when the NVIDIA prompt checkpoint is not there.

In this release

• The default model id is qwen3.8-27b on every target. It was spark on the CUDA engine and b70 on the Intel Arc Pro B70 engine. /v1/models, /health and the replies report the new id. Requests may still name any model, so existing clients keep working; FERRITE_MODEL_ID sets another id (for example FERRITE_MODEL_ID=spark to keep the old one).
• RTX 5080 preset without the NVIDIA checkpoint. PROMPT=bf16 (also the fallback when the checkpoint is missing) ran out of device memory while loading. On small-memory devices the prompt now goes through the EXL3 weights on the decode kernels.
• RTX 5060 Ti 16 GB results. The RTX 5080 build runs on an RTX 5060 Ti 16 GB: about 140 tokens per second on code and 344 on a 500-token code edit, with the prompt chunk size that suits its 8-lane link.

Details: docs/rtx-5080.md. Full list: CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.4.1-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The RTX 5080 and Arc Pro B70 targets are built from source.
• ferrite-0.4.1-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Ferrite 0.4.0

Ferrite 0.4.0 lets the Intel Arc Pro B70 engine serve up to four requests at once.

In this release

• Up to four requests at once on the Arc Pro B70 (FERRITE_SLOTS=4 with scripts/serve-xe.sh, from source). Each request gets a lane with its own conversation state, and /health reports max_batch_size 4.
• One verify window for all lanes. Lanes that decode together share a window of up to 16 rows, so the weights are read once for all of them: four lanes total about 1.4x the speed of one request (126 vs 90 tokens per second with a 2K shared prefix, 85 vs 46 with 16K).
• Shared prompt start reused. A prompt that starts like another lane's reuses that part in whole 4096-token prompt windows.
• Same text as alone. Every lane's output is what the same request gives when sent alone, with greedy decoding and with seeded sampling.
• The default is unchanged: one request at a time with 128K context. In lanes mode the context is split into one lane of 81,920 tokens and three of 28,672.

Details and the full table: docs/arc-b70.md. Full list: CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.4.0-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The DGX Spark engine itself is unchanged from 0.3.0. The RTX 5080 and Arc Pro B70 targets are built from source.
• ferrite-0.4.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Credits: CREDITS.md.

Ferrite 0.3.0

Ferrite 0.3.0 adds an experimental third target, one NVIDIA RTX 5080, and makes parallel requests that share a prefix much cheaper.

In this release

• NVIDIA RTX 5080 (experimental, from source). The CUDA engine serves Qwen3.8-27B on one 16 GB card with 128K context: a 6-bit K/V cache, NVFP4/FP8 prompt weights streamed from pinned host memory, MTP drafting with copy rows. docs/rtx-5080.md
• Up to four requests at once on the RTX 5080 from one shared pool of context (SLOTS=4): a request that fits a unit runs beside others, a longer one takes the whole pool and runs alone.
• Shared prefix computed once. Requests that arrive together and start with the same text (parallel lanes of one job, agents with one system prompt) prefill that text once. docs/prompt-cache.md
• Three or more streams decoding at once share their verify rows differently with DFlash2 drafting, which raises their total speed.
• tools/bench/lanes.py: a benchmark and gate for N requests sharing a prefix. /health field names are pinned by a test.
• Supported systems are now listed in the README: Ferrite runs on Linux only.
• Fixed: captured CUDA graphs were never released and could fill device memory with many batch shapes.

Full list: CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.3.0-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The RTX 5080 and Arc Pro B70 targets are built from source.
• ferrite-0.3.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Credits: CREDITS.md.

Ferrite 0.2.0

Ferrite 0.2.0 adds an experimental second target: the same model and server on one Intel Arc Pro B70.

In this release

• Intel Arc Pro B70 engine (experimental). fe-server --engine xe serves Qwen3.8-27B on one Arc Pro B70 (32 GB), with Rust kernels compiled to SPIR-V and loaded through Level Zero: XQ 5.0 bpw weights, int8 prompt processing, lossless DFlash2 and MTP drafting, a prefix cache and 128K context. It is built from source and serves one request at a time: docs/arc-b70.md.
• XQ model: BHCC2025/Qwen3.8-27B-XQ-5.0bpw-mix, fetched and sha256-checked by scripts/fetch-models.sh xq.
• fe-server: streamed responses declare charset=utf-8, and engines may report their own timings.
• A CHANGELOG.md.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Assets

• ferrite-0.2.0-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The Arc Pro B70 engine is not in this package; build it from source.
• ferrite-0.2.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Requirements, settings and benchmarks: docs/quickstart.md, docs/benchmarks.md. Credits: CREDITS.md.

Ferrite 0.1.0

First release of Ferrite: an LLM inference engine for NVIDIA DGX Spark, written in Rust down to the GPU kernels, serving Qwen3.8-27B behind an OpenAI-compatible API.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Or let your AI agent do it: "Install Ferrite on this DGX Spark by following https://github.com/BHCC2025/ferrite/blob/main/AGENTS.md exactly; stop and ask me if any check fails."

In this release

• One Spark, or two over RoCE (tensor parallel); 128K context; up to 4 concurrent requests.
• EXL3 3.0 bpw decode, NVFP4/FP8 prompt processing, DFlash2 speculative decoding with verify trees and copy rows (lossless).
• A prompt cache for agents: turn snapshots, parked conversations, one shared system prompt.
• Model: BHCC2025/Qwen3.8-27B-EXL3-3.0bpw-mix, fetched and sha256-checked by scripts/fetch-models.sh.

Assets

• ferrite-0.1.0-dgx-spark-aarch64.tar.gz: prebuilt binaries (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed.
• ferrite-0.1.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Requirements, settings and benchmarks: docs/quickstart.md, docs/benchmarks.md. Credits: CREDITS.md.