Skip to content

fusedtok v0.3.0

Choose a tag to compare

View all tags
@Hai-Wenxiang Hai-Wenxiang released this 23 Aug 16:20
· 7 commits to main since this release
v0.3.0
252d36e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The selection-performance and dtype release.

Changed

  • Selection sort rewritten: per-block chunk sorts + merge-path levels (barriers O(log^2 m) -> O(log nb)); the top-p nucleus count is now a grid-parallel scan (was a ~0.9 ms single-thread loop at 131k vocab)
  • bf16 elementwise dispatch tiers x8 (uint4, 16B) / x4 / scalar - GDDR7 parts gain 2.8x (RTX 5060 Ti bf16 silu 28.3 -> 10.2 us, 1.05x vs torch); Ampere unchanged at DRAM parity

Added

  • INT8 utilities: quantize_int8 / dequantize_int8 (symmetric per-tensor) and fused qadd_int8 (dequant -> add -> requant, one device pass)

Honest numbers (dual-GPU)

  • Ampere RTX 3060: topk(131k) 1.62x vs torch; topp@131k 2178 -> 1146 us
  • Blackwell RTX 5060 Ti: bare topk 0.31x vs torch's CUB (decoupled-lookback queued for v0.4); sample_topp 903 -> 672 us; argmax 1.07x

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

Assets 2
Loading