Experimental research artifact
First public release of a GPU-only Triton backend for tiiuae/Falcon3-10B-Instruct-1.58bit, integrating K-contiguous packed ternary weights, DP4A decode, StaticCache, and CUDA Graph replay.
Measured results — RTX 5070, batch 1
- 97.51 tokens/s hybrid packed decode
- 9.89 tokens/s stock Transformers BitLinear baseline
- 9.86x observed steady-state decode speedup
- Fully packed prefill: 426.63 tokens/s versus 298.72 tokens/s stock
Numerical validation
- 64/64 incremental M=1 positions had bit-exact full-vocabulary logits (131,072 logits per position)
- 24/24 greedy sequences and 1,194/1,194 generated tokens matched the stock-prefill baseline
- Corrected synthetic kernel suite: 8/8 shapes bit-exact against independent PyTorch int32 reference
Scope and limitations
Results cover one RTX 5070 and one Windows/PyTorch/Triton stack. Timings measure steady-state model execution after warmup and exclude checkpoint loading, tokenization, repack, JIT compilation, graph capture, and streaming. The baseline is stock Transformers BitLinear, not the official Microsoft kernel, BitBLAS, vLLM, or SGLang. The packed-word DP4A primitive is prior art; this artifact contributes the Triton/Falcon3/Transformers/CUDA Graph integration and measurements.
License: MIT, copyright (c) 2026 Wade Fili.
Release asset integrity
Falcon158-Triton-v0.1.0.zip
SHA-256: B12DFB8A259897BD30995238DD63B93FD2EFC87BF594E38E52A0042DFF0031D4