v0.0.10

What's Changed

This release introduces official Pyodide (PyEmscripten) support on PyPI with support for Python version 3.14 only.

• feat: add pyodide compatibility by @chonk-lain in #7
• chore: update publish workflow by @chonk-lain in #8
• chore: bump version by @chonk-lain in #9
• feat: native PyPI support for Pyodide (PyEmscripten) by @chonk-lain in #10

Full Changelog: v0.0.9...v0.0.10

v0.0.10-alpha.1

What's Changed

• feat: add pyodide compatibility by @chonk-lain in #7
• chore: update publish workflow by @chonk-lain in #8

Full Changelog: v0.0.9...v0.0.10-alpha.1

0.0.10-alpha.2

What's Changed

• chore: bump version by @chonk-lain in #9

Full Changelog: v0.0.10-alpha.1...0.0.10-alpha.2

v0.0.9

Tokie 0.0.9 🦜🎉

What's Changed

• Refactor pretokie: trait-based Core replaces 7 duplicated iterators by @chonknick in #3
• support python 3.10 by @chonk-lain in #5

New Contributors

• @chonk-lain made their first contribution in #5

Full Changelog: v0.0.8...v0.0.9

v0.0.8

What's Changed

pretokie v0.0.3

• Fix pretokenizer accuracy across all 7 specialized iterators (CL100K, O200K, GPT-2, SmolLM, DeepSeek, Qwen, Voyage)
• Improved whitespace, punctuation, and Unicode handling

tokie v0.0.8

• Add WordPiece max_input_chars_per_word limit (HF default: 100) — words exceeding this length now correctly map to [UNK]
• Parse max_input_chars_per_word from tokenizer.json
• Backward-compatible .tkz serialization (old files default to 100)

Accuracy

• 74/74 models pass token-exact accuracy tests against HuggingFace tokenizers and tiktoken-rs on 1MB enwik8
• 133/133 BERT edge-case strings pass (including long words, Unicode, control characters)

v0.0.7

What's new

• 60/60 tokenizer accuracy — fixed the last 4 failing Unigram models (deepset-mxbai, Jina v3, Cohere multilingual v3/light)
• Fixed Unigram Viterbi scoring for models without byte fallback tokens

Fix details

Unigram models without <0xXX> byte fallback tokens (like deepset-mxbai, Jina v3, Cohere multilingual) had unk_score=0.0, making <unk> "free" in Viterbi and causing incorrect segmentation. Now uses a heavy penalty (-100.0) for these models while preserving the actual model score for models with byte fallback (T5, XLM-R).

v0.0.6

What's New

22/22 tokenizer accuracy (100% match with HuggingFace)

Fixed the remaining 5 tokenizer accuracy failures, achieving perfect token-level match against HuggingFace tokenizers across all 22 tested models on 1MB of enwik8.

Fixes:

• SentencePiece BPE non-monotonic merges (Voyage-code-2, Voyage-law-2): RadixHeap now handles non-monotonic merge orderings via an overflow vector, fixing models where merge ranks aren't strictly increasing.

• Added token pre-splitting (Phi-2, ModernBERT): Non-special added tokens (multi-space/tab sequences) are now matched before pretokenization using a DAAC automaton, matching HuggingFace's behavior. Also loads added tokens from tokenizer.json when using .tkz format.

• Unigram <unk> handling (T5): Uses the actual model score for <unk> instead of hardcoded -100.0, collapses consecutive <unk> tokens into one, and advances by full UTF-8 character on unknown bytes — all matching SentencePiece behavior.

• SentencePiece normalizer: Format characters (ZWNJ, ZWJ, directional marks) are now mapped to space instead of stripped, matching SentencePiece's Precompiled charsmap.

Performance

No speed regression. tokie remains 1.8x–79x faster than HuggingFace tokenizers and 19–23x faster than tiktoken-rs:

Model	tokie	HF tokenizers	Speedup
BERT	379 MB/s	4.8 MB/s	79x
GPT-2	321 MB/s	5.5 MB/s	58x
Llama-3.2	339 MB/s	6.4 MB/s	53x
cl100k	208 MB/s	10.9 MB/s (tiktoken)	19x
T5 (Unigram)	18.3 MB/s	4.3 MB/s	4.3x
XLM-R (SP-BPE)	17.7 MB/s	4.2 MB/s	4.2x

New

• bench_vs_hf example: comparison benchmark against HF tokenizers and tiktoken-rs

v0.0.5

What's New

Features

• encode_with_offsets() — byte offset tracking for each token in the normalized input, with parallel encoding for large texts
• num_special_tokens_to_add() — HF-compatible API for querying special token count
• Decoder refactor — Decoder wraps VocabDecoder + DecoderType for cleaner architecture

Improvements

• Package discoverability — PyPI keywords/classifiers, crates.io keywords/categories, docs.rs badge
• README on crates.io — now displays the full README
• Python benchmark suite — scripts/benchmark_vs_hf.py for reproducible comparisons
• README — Python benchmark table showing 10-136x speedups across 5 models

Cleanup

• Removed unused SentencePieceBPEv2 (400 lines of dead code)
• 33 Python tests (up from 27), including unigram (T5, XLM-R) coverage

Benchmarks (Python, 45KB text)

Model	Speedup vs HF
BERT (WordPiece)	61x
GPT-2 (BPE)	50x
Llama 3 (BPE)	54x
Qwen 3 (BPE)	54x
Gemma 3 (SentencePiece)	10x
T5 (Unigram)	19x

v0.0.4

What's New

• Vocab access APIs: id_to_token(), token_to_id(), get_vocab(), decode_batch() — in both Rust and Python
• Padding & truncation: enable_padding(), enable_truncation(), no_padding(), no_truncation() — HuggingFace-compatible API
• Encode returns Encoding: encode() and encode_batch() now return Encoding objects with ids, attention_mask, and type_ids
• Batch pair encoding: encode_pair_batch() for parallel cross-encoder inference
• 60 pre-built tokenizers: Published to tokiers HuggingFace org
• Performance: Cached num_cpus(), zero-copy id_to_token() via Cow<str>
• Cleanup: Removed 27 redundant examples, 10 unused dev-dependencies, -4500 lines net

v0.0.3

What's New

• Python bindings via PyO3/maturin (pip install tokie)
• Workspace restructure — crates/ directory with tokie (Rust lib) and tokie-python
• License change — AGPL-3.0 → MIT OR Apache-2.0
• Release CI — automated publish to crates.io and PyPI on release
• Version script — scripts/bump-version.sh for synchronized version updates
• Assets — tokie bird mascot and banner images

Python API

import tokie

t = tokie.Tokenizer.from_pretrained("bert-base-uncased")
tokens = t.encode("Hello, world!")
text = t.decode(tokens)