Skip to content

fusedtok v0.2.0

Choose a tag to compare

View all tags
@Hai-Wenxiang Hai-Wenxiang released this 23 Aug 12:07
· 10 commits to main since this release
v0.2.0
0c3cdfc

v0.2.0 - the performance and dtype release.

Added

  • bfloat16 on the zero-copy torch path across the inference core (f32 compute, boundary conversion; norm weights auto-upcast)
  • sample_topp: fused nucleus sampling - softmax -> top-p -> seeded inverse-CDF draw in one cooperative kernel (deterministic per seed)

Changed

  • top-k / top-p rewritten as a single cooperative radix kernel: two-stage histogram, shared/global bitonic sort, exact tie determinism. top-k@131k: 0.77x -> 1.4x vs PyTorch (Ampere); top-p 26x faster than v0.1
  • softmax: register-resident single-read kernel, 2.6x vs torch on Blackwell, 1.0-1.1x on Ampere

Verified

  • Linux/Blackwell validation (RTX 5060 Ti, sm_120): 134 tests green, RMSNorm 3.3x / RoPE 8.3x vs torch cu128; PyPI sm_86 build JIT-verified on sm_120
  • CUDA graph capture+replay for elementwise / norms / softmax / top-k / RoPE
  • sdist pre-verified (twine check, UTF-8 audit) before publishing

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

Assets 2
Loading