Repository navigation
Ferrite 0.3.0
Ferrite 0.3.0 adds an experimental third target, one NVIDIA RTX 5080, and makes parallel requests that share a prefix much cheaper.
In this release
- NVIDIA RTX 5080 (experimental, from source). The CUDA engine serves Qwen3.8-27B on one 16 GB card with 128K context: a 6-bit K/V cache, NVFP4/FP8 prompt weights streamed from pinned host memory, MTP drafting with copy rows. docs/rtx-5080.md
- Up to four requests at once on the RTX 5080 from one shared pool of context (
SLOTS=4): a request that fits a unit runs beside others, a longer one takes the whole pool and runs alone. - Shared prefix computed once. Requests that arrive together and start with the same text (parallel lanes of one job, agents with one system prompt) prefill that text once. docs/prompt-cache.md
- Three or more streams decoding at once share their verify rows differently with DFlash2 drafting, which raises their total speed.
tools/bench/lanes.py: a benchmark and gate for N requests sharing a prefix./healthfield names are pinned by a test.- Supported systems are now listed in the README: Ferrite runs on Linux only.
- Fixed: captured CUDA graphs were never released and could fill device memory with many batch shapes.
Full list: CHANGELOG.md.
Install on a DGX Spark (one line):
curl -fsSL https://raw.githubusercontent.com/BHCC2025/ferrite/main/scripts/install.sh | bashAssets
ferrite-0.3.0-dgx-spark-aarch64.tar.gz: prebuilt binaries for DGX Spark (aarch64, CUDA 13, kernels for sm_121a) plus scripts and docs. No Rust toolchain needed. The RTX 5080 and Arc Pro B70 targets are built from source.ferrite-0.3.0-dgx-spark-aarch64.tar.gz.sha256: its checksum (sha256sum -c).
Credits: CREDITS.md.