fusedtok v0.3.1

Maintenance patch over 0.3.0 - docs and demo only, no kernel changes.

Fixed

• README operator table synced with the shipped 0.3.0 surface (INT8 row, top-k 1.6x, stale roadmap pointer) - bilingual

Changed

• demo gains the INT8 tour: roundtrip error bound, zero-copy scale parity, fused qadd vs an explicit float reference
• dequantize_int8 dispatches through the shared device-path helper

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

fusedtok v0.3.0

The selection-performance and dtype release.

Changed

• Selection sort rewritten: per-block chunk sorts + merge-path levels (barriers O(log^2 m) -> O(log nb)); the top-p nucleus count is now a grid-parallel scan (was a ~0.9 ms single-thread loop at 131k vocab)
• bf16 elementwise dispatch tiers x8 (uint4, 16B) / x4 / scalar - GDDR7 parts gain 2.8x (RTX 5060 Ti bf16 silu 28.3 -> 10.2 us, 1.05x vs torch); Ampere unchanged at DRAM parity

Added

• INT8 utilities: quantize_int8 / dequantize_int8 (symmetric per-tensor) and fused qadd_int8 (dequant -> add -> requant, one device pass)

Honest numbers (dual-GPU)

• Ampere RTX 3060: topk(131k) 1.62x vs torch; topp@131k 2178 -> 1146 us
• Blackwell RTX 5060 Ti: bare topk 0.31x vs torch's CUB (decoupled-lookback queued for v0.4); sample_topp 903 -> 672 us; argmax 1.07x

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

fusedtok v0.2.1

Patch release over 0.2.0 - no kernel changes.

Fixed

• sample_topp missing from __all__ (dropped by star-imports)
• demo import fallback (dev trees now work without PYTHONPATH setup); demo tours fused sampling + bf16

Changed

• dead functor removed (nvcc warning on every Linux build)
• per-device benchmark artifacts (docs/benchmark_rt{3060,5060ti}.json)

Added

• README minimal per-token sampling loop (EN + ??)

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

fusedtok v0.2.0

v0.2.0 - the performance and dtype release.

Added

• bfloat16 on the zero-copy torch path across the inference core (f32 compute, boundary conversion; norm weights auto-upcast)
• sample_topp: fused nucleus sampling - softmax -> top-p -> seeded inverse-CDF draw in one cooperative kernel (deterministic per seed)

Changed

• top-k / top-p rewritten as a single cooperative radix kernel: two-stage histogram, shared/global bitonic sort, exact tie determinism. top-k@131k: 0.77x -> 1.4x vs PyTorch (Ampere); top-p 26x faster than v0.1
• softmax: register-resident single-read kernel, 2.6x vs torch on Blackwell, 1.0-1.1x on Ampere

Verified

• Linux/Blackwell validation (RTX 5060 Ti, sm_120): 134 tests green, RMSNorm 3.3x / RoPE 8.3x vs torch cu128; PyPI sm_86 build JIT-verified on sm_120
• CUDA graph capture+replay for elementwise / norms / softmax / top-k / RoPE
• sdist pre-verified (twine check, UTF-8 audit) before publishing

Full changelog: https://github.com/Hai-Wenxiang/fusedtok/blob/main/CHANGELOG.md

fusedtok v0.1.2

Documentation-only patch over 0.1.0/0.1.1.

Fixed

• The 0.1.1 PyPI description rendered as mojibake: the README link rewrite was performed with a GBK-default console reader, corrupting every non-ASCII character. Both READMEs are restored and verified (twine check PASSED; sdist PKG-INFO is clean UTF-8 with Chinese intact).
• en/zh README cross-links pointed at the wrong files; corrected.

See the changelog.

fusedtok v0.1.1

fusedtok v0.1.1

Documentation-only patch over v0.1.0.

Fixed

• PyPI page links: relative README links (Chinese README, benchmark chart, community docs) resolved against pypi.org and returned 404. All repo-file links are now absolute GitHub URLs.

See the changelog for details.

fusedtok v0.1.0

First public release of fusedtok - fused CUDA kernels for LLM inference,
with numpy/torch tensor API and a zero-copy CUDA path.

Highlights

• 19 operators: RMSNorm (+fused residual), LayerNorm, RoPE (interleaved
  and NeoX layouts, kv-cache pos_offset), SwiGLU, row-wise softmax,
  SiLU / GeLU (erf + tanh) / ReLU / Tanh / Sigmoid, add / mul, temperature,
  repetition penalty, top-k, top-p (nucleus), argmax
• Three execution paths per op: CPU reference (ground truth, runs
  without a GPU), staged CUDA (numpy in / out), and zero-copy CUDA - kernels
  run directly in torch device buffers via data_ptr(), async and
  stream-ordered with torch ops
• Optimized kernels: block-per-row reductions, float4-vectorized
  elementwise, exp2f/sincosf RoPE, parallel packed-key selection with
  deterministic ties (cooperative-groups, host-loop fallback)
• Benchmarks (RTX 3060, float32, CUDA events): RoPE 6.2x vs PyTorch
  eager, RMSNorm+residual 2.1x, SwiGLU 1.7x, argmax 1.3x;
  elementwise ops at bandwidth parity (~330-500 GB/s). Full data:
  docs/benchmark_results.json, reproduce with python benchmarks/bench.py
• Quality: 95 pytest cases (formula parity, edge cases, error contract,
  GPU-vs-CPU), CI on GitHub Actions, scikit-build-core packaging
  (sm_80/sm_86 cubins + compute_86 PTX for newer GPUs)

Install

pip install from source (prebuilt wheels planned):

pip install git+https://github.com/Hai-Wenxiang/fusedtok.git@v0.1.0

Requires CUDA Toolkit >= 12.0 and an RTX 30-series / Ampere or newer GPU.

Artifacts attached: source tarball and a cp312 Windows wheel built on the
reference machine (RTX 3060, CUDA 13.3, Python 3.12).