Show and Tell: Falcon158-Triton — 97.5 tok/s Falcon3-10B 1.58-bit decode on RTX 5070

[OCV-Researcher]

Hi BitNet community — I’m Wade Fili. I built an experimental GPU-only Triton inference backend for tiiuae/Falcon3-10B-Instruct-1.58bit and I’m looking for independent reproductions and kernel feedback.

What is implemented

• K-contiguous packed ternary weights
• packed-word DP4A decode path
• Triton kernels integrated with Transformers
• StaticCache and CUDA Graph replay

Measured result

Single NVIDIA RTX 5070, batch 1, steady-state model execution after warmup:

Path	Throughput
Hybrid packed decode	97.51 tok/s
Stock Transformers BitLinear baseline	9.89 tok/s
Observed decode speedup	9.86x
Fully packed prefill	426.63 tok/s
Stock prefill	298.72 tok/s

Numerical validation

• 64/64 incremental M=1 positions produced bit-exact full-vocabulary logits (131,072 logits per position)
• 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline
• corrected synthetic kernel suite: 8/8 shapes bit-exact against an independent PyTorch int32 reference

Scope / caveats

• The comparison baseline is unmodified Transformers BitLinear, not Microsoft’s official GPU kernel, BitBLAS, vLLM, or SGLang.
• Timings exclude checkpoint loading, tokenization, weight repacking, JIT compilation, graph capture, and streaming.
• So far this is one GPU and one Windows/PyTorch/Triton stack.
• Packed-word DP4A is prior art; the artifact contribution is the Falcon3/Transformers/Triton/CUDA Graph integration and the accompanying measurements.

Repository: https://github.com/OCV-Researcher/Falcon158-Triton

Release v0.1.0: https://github.com/OCV-Researcher/Falcon158-Triton/releases/tag/v0.1.0

I would especially appreciate:

1. reproductions on Ampere, Hopper, Ada, or other Blackwell GPUs;
2. comparisons against the official BitNet GPU path and other low-bit runtimes;
3. feedback on the packed GEMV/decode bottlenecks and benchmark methodology.

The code is MIT licensed. Raw methodology, limitations, and numerical checks are documented in the repository.