v0.0.8

What's Changed

pretokie v0.0.3

• Fix pretokenizer accuracy across all 7 specialized iterators (CL100K, O200K, GPT-2, SmolLM, DeepSeek, Qwen, Voyage)
• Improved whitespace, punctuation, and Unicode handling

tokie v0.0.8

• Add WordPiece max_input_chars_per_word limit (HF default: 100) — words exceeding this length now correctly map to [UNK]
• Parse max_input_chars_per_word from tokenizer.json
• Backward-compatible .tkz serialization (old files default to 100)

Accuracy

• 74/74 models pass token-exact accuracy tests against HuggingFace tokenizers and tiktoken-rs on 1MB enwik8
• 133/133 BERT edge-case strings pass (including long words, Unicode, control characters)