What's New
- Python bindings via PyO3/maturin (
pip install tokie)
- Workspace restructure — crates/ directory with
tokie (Rust lib) and tokie-python
- License change — AGPL-3.0 → MIT OR Apache-2.0
- Release CI — automated publish to crates.io and PyPI on release
- Version script —
scripts/bump-version.sh for synchronized version updates
- Assets — tokie bird mascot and banner images
Python API
import tokie
t = tokie.Tokenizer.from_pretrained("bert-base-uncased")
tokens = t.encode("Hello, world!")
text = t.decode(tokens)