v0.0.5

What's New

Features

• encode_with_offsets() — byte offset tracking for each token in the normalized input, with parallel encoding for large texts
• num_special_tokens_to_add() — HF-compatible API for querying special token count
• Decoder refactor — Decoder wraps VocabDecoder + DecoderType for cleaner architecture

Improvements

• Package discoverability — PyPI keywords/classifiers, crates.io keywords/categories, docs.rs badge
• README on crates.io — now displays the full README
• Python benchmark suite — scripts/benchmark_vs_hf.py for reproducible comparisons
• README — Python benchmark table showing 10-136x speedups across 5 models

Cleanup

• Removed unused SentencePieceBPEv2 (400 lines of dead code)
• 33 Python tests (up from 27), including unigram (T5, XLM-R) coverage

Benchmarks (Python, 45KB text)

Model	Speedup vs HF
BERT (WordPiece)	61x
GPT-2 (BPE)	50x
Llama 3 (BPE)	54x
Qwen 3 (BPE)	54x
Gemma 3 (SentencePiece)	10x
T5 (Unigram)	19x