Skip to content

v0.0.6

Choose a tag to compare

View all tags
@chonknick chonknick released this 26 Mar 02:11
v0.0.6
58ec637

What's New

22/22 tokenizer accuracy (100% match with HuggingFace)

Fixed the remaining 5 tokenizer accuracy failures, achieving perfect token-level match against HuggingFace tokenizers across all 22 tested models on 1MB of enwik8.

Fixes:

  • SentencePiece BPE non-monotonic merges (Voyage-code-2, Voyage-law-2): RadixHeap now handles non-monotonic merge orderings via an overflow vector, fixing models where merge ranks aren't strictly increasing.

  • Added token pre-splitting (Phi-2, ModernBERT): Non-special added tokens (multi-space/tab sequences) are now matched before pretokenization using a DAAC automaton, matching HuggingFace's behavior. Also loads added tokens from tokenizer.json when using .tkz format.

  • Unigram <unk> handling (T5): Uses the actual model score for <unk> instead of hardcoded -100.0, collapses consecutive <unk> tokens into one, and advances by full UTF-8 character on unknown bytes — all matching SentencePiece behavior.

  • SentencePiece normalizer: Format characters (ZWNJ, ZWJ, directional marks) are now mapped to space instead of stripped, matching SentencePiece's Precompiled charsmap.

Performance

No speed regression. tokie remains 1.8x–79x faster than HuggingFace tokenizers and 19–23x faster than tiktoken-rs:

Model tokie HF tokenizers Speedup
BERT 379 MB/s 4.8 MB/s 79x
GPT-2 321 MB/s 5.5 MB/s 58x
Llama-3.2 339 MB/s 6.4 MB/s 53x
cl100k 208 MB/s 10.9 MB/s (tiktoken) 19x
T5 (Unigram) 18.3 MB/s 4.3 MB/s 4.3x
XLM-R (SP-BPE) 17.7 MB/s 4.2 MB/s 4.2x

New

  • bench_vs_hf example: comparison benchmark against HF tokenizers and tiktoken-rs
Assets 2
Loading