Show and Tell: Hyphae BitNet Runtime - exact CPU inference, C ABI, serving, Bonsai Q1, and reproducible evidence

[Mario Gutierrez (terrizoaguimor)]

Hi BitNet community,

We are sharing Hyphae BitNet Runtime, an open-source fork of Microsoft BitNet focused on correctness-first CPU inference, an embeddable runtime/API, and reproducible numerical and performance evidence.

The project started from Microsoft BitNet commit 0b341e5, with the corresponding llama.cpp engine now vendored in-tree and pinned by both commit and exact Git tree hash. Upstream imports are reviewed manually and must pass the numerical and hardware validation gates.

Repository: https://github.com/Hyphae-Research-Foundation/hyphae-bitnet

Latest release: https://github.com/Hyphae-Research-Foundation/hyphae-bitnet/releases/tag/v0.3.3

The current source/package version is 0.3.2. Hyphae/Celiums contributions are Apache-2.0; retained upstream code keeps its original MIT/BSD and third-party licensing. This is an independent community project and is not affiliated with, sponsored by, endorsed by, or supported by Microsoft.

What changed from the upstream baseline

1. A strict and documented I2_S numerical contract

We made the BitNet b1.58 2B I2_S path explicit and testable rather than relying on implementation assumptions:

• canonical 128-coefficient ternary packing (-1 -> 0, 0 -> 1, +1 -> 2);
• tensor-wide F32 weight scaling;
• per-row I8_S activation quantization with round-half-away-from-zero and activation-code sums;
• exact unsigned-code offset recovery, y = (D - S) rho;
• correct zero-activation behavior and accumulator bounds;
• fixes for scalar packing/dot semantics and non-VNNI overflow;
• squared-ReLU for the BitNet b1.58 feed-forward graph;
• native/VNNI, portable AVX2-without-VNNI, and scalar reference paths.

The equations, byte layout, bounds, and non-claims are documented in the numerical contract. Exact epilogues and Q1 packing/correction live in the small in-tree celiums-exact module, which is called by both kernels and test oracles so the tests do not silently reimplement different math.

We also added structural GGUF validation for architecture, file type, required projections, dimensions, packed byte counts, and finite scales.

2. A real runtime/product surface instead of a llama-cli wrapper

The public binary is hyphae-bitnet (celiums-bitnet remains as a compatibility alias) with:

• run: one-shot generation;
• serve: native HTTP completion server;
• bench: JSONL prefill/decode evidence;
• validate: checked model loading;
• version: product, engine, tree, profile, and strictness provenance.

There is an experimental C ABI with opaque Runtime, Model, Session, and Request handles. It includes tokenization/detokenization, prefill, single-token decode, copied logits, sampling, synchronous streaming callbacks, stop sequences, cooperative cancellation, and model/runtime provenance without exposing llama.cpp or GGML types.

The HTTP server implements a deliberately limited OpenAI-shaped completions/chat subset with SSE, health/model endpoints, Prometheus metrics, body/token/concurrency limits, and cancellation on client disconnect. Non-loopback binding is refused unless authentication is configured or an explicit unsafe override is supplied.

We also repaired an ABI mistake made during 0.3.1 development: the released v0.3.0 option layouts and by-value return convention are frozen, while newer RAM/layout/family/sequence controls use sized _ex structs and caller-buffer initializers. A frozen v0.3.0 client is compiled and run in CI against the current shared library.

3. RAM is an explicit serving lever, with fail-closed accounting

Packed GGUF remains the durable store. Optional ISA-specific compute layouts are materialized as a bounded working set, not as an unaccounted copy:

• one aggregate runtime RAM reservation ledger;
• automatic safe budget or explicit --ram-budget-bytes;
• model load/session creation fail with RAM_BUDGET_EXCEEDED before crossing the cap;
• compute-layout and session/KV estimates are exposed through the extended C API;
• n_seq > 1 is rejected because decode currently uses sequence 0 only rather than pretending to support safe multi-sequence serving.

4. Exact Q1 CPU work for Bonsai 27B

In addition to the certified BitNet 2B I2_S target, the runtime has an explicit opt-in CPU text family for the pre-quantized Bonsai 27B Q1_0 model (qwen35, file type 40):

• exact Q1 correction, dot = d_w d_x (2P - S);
• x86 bit-packed 4x8 VNNI panels;
• ARM packed 4x8 panels with direct sign decode into DOTPROD/I8MM tiles;
• eight-row prefill GEMM tiles that reuse each Q1 weight panel before rereading it;
• execution-scoped exact Q8 activation packing reuse for eligible repeated projections;
• packed/layout/generic oracle coverage;
• an experimental Neoverse V2 SVE2 VL=128 decode path.

Whole-model testing showed the SVE2 prototype was exact but slower than the NEON path, so it remains opt-in rather than being promoted based on an isolated kernel result. Likewise, Q8 activation reuse was exact but neutral at the measured whole-model scale. We retained the evidence and the conservative default.

We intentionally do not ship LUT-GEMM or T-MAC in the strict paths because those methods are approximate. The goal here is exact integer accumulation first.

5. Optional local AI gateway, kept outside the inference process

An optional Rust gateway composes the native loopback server with authenticated Hyphae UDS and provides bounded local RAG/memory, BM25 and vector/ANN/hybrid retrieval, semantic cache, durable generation receipts, proof verification, dataset/artifact/lineage/evaluation registries, and a separate MCP stdio adapter.

The gateway is process-separated: Hyphae is not linked into GGML or the inference C ABI.

6. Reproducibility and release engineering

The repository now includes:

• a complete vendored engine tree, so a normal clone does not depend on a private or archived submodule;
• exact product commit, engine commit, and engine tree provenance;
• full-vocabulary logits captures/comparison with a bitwise release mode;
• scalar, AVX2-without-VNNI, native, ASan/UBSan, package, gateway, and native ARM64 CI;
• frozen-header ABI testing and installed CMake consumer testing;
• Linux x86-64 native, avx2, and scalar release archives;
• machine-readable manifests with compiler, Rust/Hyphae pin, platform, source cleanliness, and SHA-256 evidence;
• raw benchmark receipts committed under docs/ instead of only headline numbers.

Models and validation scope

Family	Model/format	Status
BitNet	microsoft/BitNet-b1.58-2B-4T, I2_S	Certified target; pinned revision and validated GGUF SHA-256
Bonsai	Bonsai-27B-Q1_0.gguf, Q1_0	Explicit CPU text family; pre-quantized model required

Model files are not redistributed. Production users should pin the publisher revision and model digest.

A few measured results, with caveats

These are hardware-specific historical receipts, not universal claims:

• Xeon 8559C, Bonsai 27B Q1_0, pp128: changing the exact Q1 GEMM from four-row to eight-row reuse improved prefill from 2.389 to 2.593 tok/s at one thread (+8.6%) and 75.41 to 80.40 tok/s at 48 threads (+6.6%). Decode was intentionally unchanged and remained bandwidth-bound.
• The same Xeon with the certified BitNet 2B I2_S model measured 77.82 pp128 / 9.58 tg128 tok/s at one thread and 1304 / 153 tok/s at 48 threads. This is a different model and roofline, included to avoid conflating 2B I2_S with 27B Q1.
• Graviton 4, Bonsai 27B Q1_0, the latest 96-thread baseline receipt measured 124.62 pp128 / 21.56 tg128 tok/s. That receipt used one repetition; it is retained as implementation evidence, not presented as a statistical benchmark study.
• A deliberately impossible 64-byte RAM budget fails closed before model allocation on the measured x86 and ARM hosts.

Raw receipts and methodology are linked from the README and the vendored CPU hot-path review.

Current limits

To keep the claims precise, the supported product is currently:

• CPU-only;
• official release archives for Linux x86-64; ARM64 is source-tested/measured but not an official package target;
• no arbitrary llama.cpp models or generic Q4 files;
• no Prism Q2_0, GPU offload, vision, speculative decoding, or continuous batching;
• no claim that the OpenAI-shaped HTTP subset implements the complete OpenAI API;
• no claim of numerical identity between different model families.

Where community contributions would help most

We would especially value:

1. independent reproduction of I2_S full-vocabulary logits and the benchmark receipts on additional x86 and ARM CPUs;
2. review of the I2_S/Q1 packing, correction, accumulator bounds, and zero-row behavior;
3. better exact packed GEMV/GEMM kernels, cache blocking, activation reuse, and phase-aware scheduling without relaxing the numerical contract;
4. safe multi-sequence/continuous-batching designs for the public Runtime API;
5. packaging and validation on additional Linux distributions and ARM64 systems;
6. security review of the C ABI lifetime rules, RAM ledger, native server, and optional gateway;
7. narrowly scoped fixes that should be proposed back to Microsoft BitNet or llama.cpp.

The detailed history is in CHANGES.md, and the repository contains the tests, benchmark JSON, threat model, release manifests, and exactness documentation needed to challenge the claims.

Feedback, independent measurements, bug reports, and pull requests are very welcome. If anyone is interested in reproducing a specific kernel or model-backed gate, we can help identify the exact model revision, digest, build profile, and evidence command.