Skip to content

FV5 (Fermion Neutrino) — Blackwell sm_120 — fv5-b1

Latest
Latest

Choose a tag to compare

View all tags
@Andgihat Andgihat released this 29 Jul 13:53
fv5-b1
916d6da

First build of the FV5 port for BeeLlama 4.2 — runs Fermion Research's five-value ternary
format (Neutrino-1 8B, a Qwen3-8B quant) natively on NVIDIA Blackwell (sm_120), plus a
decode-kernel optimization. Weights are Fermion's (Apache-2.0); this adds the runtime support and
a faster on-disk layout. Numerics are bit-identical to Fermion's reconstruction.

What this branch (v0.4.2-fv5) adds

  • FV5 / FV5B quant types — CPU + CUDA reconstruction, f32-exact (raw-f32 activations, no MMQ / dp4a, TF32 off).
  • Co-located "v2" plane layout — the 3 bit-planes of each 8-weight group are packed into one
    uchar4 so a single coalesced load fetches all planes → DRAM utilization 63 % → ~100 % →
    +1.23× decode on the heavy FFN tensor. Output bit-identical to the standard layout.
  • Batched verify kernel (small ne11) for speculative decoding, bit-identical to single-token decode.

Results — Neutrino-1 8B, RTX 5060 Ti 16 GB, sm_120, CUDA 12.8

workload throughput
decode (single token), v2 layout 84 t/s
decode, standard SoA layout 71 t/s
decode, initial port 40 t/s
prefill (1659-token prompt) 841 t/s
speculative (0.6B draft) 61 t/s — below plain decode, see note

Speculative-decoding note: it works with Fermion's Neutrino-0.6B drafter, but for f32-exact
FV5 it does not beat plain decode — batched verify can't use the cheap int8/MMQ path that makes
drafting pay off for normal quants, so it stays only ~1.09× faster per token. Documented, not a bug.

Model

GGUF: https://huggingface.co/Andgihat/Neutrino-8B-FV5-BeeLlama — use neutrino-8b-fv5-v2.gguf.

⚠️ Requirements

  • NVIDIA Blackwell (RTX 50-series), sm_120 ONLY — this build is compiled for
    CMAKE_CUDA_ARCHITECTURES=120 and will not run on Ada / Ampere / older GPUs.
  • CUDA 12.8 runtime — cudart64_12.dll, cublas64_12.dll, cublasLt64_12.dll on PATH
    (install CUDA Toolkit 12.8, or place them next to the exe). cuBLAS is used for prefill.

Run

llama-completion.exe -m neutrino-8b-fv5-v2.gguf -ngl 99 --temp 0 -no-cnv -p "The capital of France is"

Recommended sampling (Fermion): --temp 0…0.01, --repeat-penalty 1.05. Full run notes in RUN.md inside the asset.

Build from source

cmake -B build -G Ninja -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_FORCE_MMQ=ON
cmake --build build --target llama-completion -j

Credits

Model & FV5 format © Fermion Research (Neutrino-1 8B, Apache-2.0).
Runtime: BeeLlama / llama.cpp.

Assets 3
Loading