Skip to content

v0.13.0 — int8 quantized inference

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 07 Jul 01:36
v0.13.0
4e21e1e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

zenith.quantize adds weight-only int8 quantization: QuantizedLinear + quantize_int8(model), ~4× smaller linear weights with greedy output unchanged on the 10.7M model. zenith generate --int8. Saves memory (no int8 matmul kernel, so not a CPU/MPS speedup) — honest scope.

Assets 2
Loading