v0.4.2

[0.4.2] 2026-11-09

Bug fixes

• The Rust core and nested packages report the release version (#11)

Documentation

• Diagrams on white in the shared theme, make diagram for every diagram, and a self-contained Definition of Done (#9)

Build and CI

• Build Linux and macOS wheels with the Rust core and the sdist for every release (#10)

v0.4.1

[0.4.1] 2026-11-08

Bug fixes

• Report the released version and keep version strings in step on release (#8)

Documentation

• Describe the supply chain accurately (#5)
• Complete citation metadata and bump it on release (#7)

Maintenance

• Latest toolchain and CI versions (#6)

v0.4.0

[0.4.0] 2026-11-02

Features

• Build the Rust core into the package and use it by default (#4)

v0.3.0

[0.3.0] 2026-11-02

Performance

• Heap-based trainer, parallel encode_batch (#3)

Documentation

• Horizontal layered architecture diagram
• Compact horizontal architecture diagram
• Remove authorship statements from docs and scripts
• Recorded demos of more commands, with their vhs tapes (#1)
• Recorded demos of the HTTP service and more commands (#2)

Maintenance

• Add release automation, AGENTS.md, llms.txt and contributor docs
• Classify squash merges in the changelog by the commits they contain

v0.2.0

Version 0.2.0 adds an optional Rust core.

• Rust implementation of the pre-tokenizer, byte map, merge loop, and trainer, exposed through PyO3 as the bpe_tokenizer_rs extension.
• BPETokenizer accepts backend="rust" and falls back to the Python implementation when the extension is not installed. The default API is unchanged.
• Parity tests compare both backends on 12,223 seeded strings and four merge sets with zero mismatches, plus every Unicode scalar value through the pre-tokenizer.
• benchmarks/compare.py measures encode throughput for Python, Rust, and tiktoken (different vocabulary, throughput reference only). On an Apple M1 the Rust backend is 13 to 19 times faster than Python.
• CI adds a rust job: cargo fmt, clippy, cargo test, maturin develop, and parity tests.

Assets: source distribution, and a macOS arm64 wheel of the Rust extension (abi3, Python 3.11 and newer).