Repository navigation

v0.3.0

HFooladi released this 18 May 10:43

· 27 commits to main since this release

3365cdc

rustmolbpe v0.3.0

Four-tokenizer ladder: CharTokenizer, AtomTokenizer, CharBPETokenizer, SmilesTokenizer
HuggingFace tokenizer.json interop (save_huggingface / from_huggingface)
Granularity-aware pickle format (v2); legacy v1 pickles still load
tokenizer_stats.py statistics script
Runtime rustmolbpe.version from CARGO_PKG_VERSION

See CHANGELOG.md for full details.

Assets 2