Release fusedtok v0.1.0 · Hai-Wenxiang/fusedtok

First public release of fusedtok - fused CUDA kernels for LLM inference,
with numpy/torch tensor API and a zero-copy CUDA path.

Highlights

19 operators: RMSNorm (+fused residual), LayerNorm, RoPE (interleaved
and NeoX layouts, kv-cache pos_offset), SwiGLU, row-wise softmax,
SiLU / GeLU (erf + tanh) / ReLU / Tanh / Sigmoid, add / mul, temperature,
repetition penalty, top-k, top-p (nucleus), argmax
Three execution paths per op: CPU reference (ground truth, runs
without a GPU), staged CUDA (numpy in / out), and zero-copy CUDA - kernels
run directly in torch device buffers via data_ptr(), async and
stream-ordered with torch ops
Optimized kernels: block-per-row reductions, float4-vectorized
elementwise, exp2f/sincosf RoPE, parallel packed-key selection with
deterministic ties (cooperative-groups, host-loop fallback)
Benchmarks (RTX 3060, float32, CUDA events): RoPE 6.2x vs PyTorch
eager, RMSNorm+residual 2.1x, SwiGLU 1.7x, argmax 1.3x;
elementwise ops at bandwidth parity (~330-500 GB/s). Full data:
docs/benchmark_results.json, reproduce with python benchmarks/bench.py
Quality: 95 pytest cases (formula parity, edge cases, error contract,
GPU-vs-CPU), CI on GitHub Actions, scikit-build-core packaging
(sm_80/sm_86 cubins + compute_86 PTX for newer GPUs)

pip install from source (prebuilt wheels planned):

pip install git+https://github.com/Hai-Wenxiang/fusedtok.git@v0.1.0

Requires CUDA Toolkit >= 12.0 and an RTX 30-series / Ampere or newer GPU.

Artifacts attached: source tarball and a cp312 Windows wheel built on the
reference machine (RTX 3060, CUDA 13.3, Python 3.12).