Skip to content

v0.2.0

Choose a tag to compare

@adeoluo adeoluo released this 02 Nov 05:03
· 20 commits to main since this release

Version 0.2.0 adds an optional Rust core.

  • Rust implementation of the pre-tokenizer, byte map, merge loop, and trainer, exposed through PyO3 as the bpe_tokenizer_rs extension.
  • BPETokenizer accepts backend="rust" and falls back to the Python implementation when the extension is not installed. The default API is unchanged.
  • Parity tests compare both backends on 12,223 seeded strings and four merge sets with zero mismatches, plus every Unicode scalar value through the pre-tokenizer.
  • benchmarks/compare.py measures encode throughput for Python, Rust, and tiktoken (different vocabulary, throughput reference only). On an Apple M1 the Rust backend is 13 to 19 times faster than Python.
  • CI adds a rust job: cargo fmt, clippy, cargo test, maturin develop, and parity tests.

Assets: source distribution, and a macOS arm64 wheel of the Rust extension (abi3, Python 3.11 and newer).