Repository navigation
v0.2.0
Version 0.2.0 adds an optional Rust core.
- Rust implementation of the pre-tokenizer, byte map, merge loop, and trainer, exposed through PyO3 as the bpe_tokenizer_rs extension.
- BPETokenizer accepts backend="rust" and falls back to the Python implementation when the extension is not installed. The default API is unchanged.
- Parity tests compare both backends on 12,223 seeded strings and four merge sets with zero mismatches, plus every Unicode scalar value through the pre-tokenizer.
- benchmarks/compare.py measures encode throughput for Python, Rust, and tiktoken (different vocabulary, throughput reference only). On an Apple M1 the Rust backend is 13 to 19 times faster than Python.
- CI adds a rust job: cargo fmt, clippy, cargo test, maturin develop, and parity tests.
Assets: source distribution, and a macOS arm64 wheel of the Rust extension (abi3, Python 3.11 and newer).