Skip to content

Ferrite 0.1.0

Choose a tag to compare

View all tags
@BHCC2025 BHCC2025 released this 09 Nov 01:11
· 4 commits to main since this release
v0.1.0
e24adf2

First release of Ferrite: an LLM inference engine for NVIDIA DGX Spark, written in Rust down to the GPU kernels, serving Qwen3.8-27B behind an OpenAI-compatible API.

Install on a DGX Spark (one line):

curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bash

Or let your AI agent do it: "Install Ferrite on this DGX Spark by following https://github.com/BHCC2025/ferrite/blob/main/AGENTS.md exactly; stop and ask me if any check fails."

In this release

  • One Spark, or two over RoCE (tensor parallel); 128K context; up to 4 concurrent requests.
  • EXL3 3.0 bpw decode, NVFP4/FP8 prompt processing, DFlash2 speculative decoding with verify trees and copy rows (lossless).
  • A prompt cache for agents: turn snapshots, parked conversations, one shared system prompt.
  • Model: BHCC2025/Qwen3.8-27B-EXL3-3.0bpw-mix, fetched and sha256-checked by scripts/fetch-models.sh.

Assets

  • ferrite-0.1.0-dgx-spark-aarch64.tar.gz: prebuilt binaries (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed.
  • ferrite-0.1.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).

Requirements, settings and benchmarks: docs/quickstart.md, docs/benchmarks.md. Credits: CREDITS.md.

Assets 4
Loading