Releases: SMC17/tokenizers-zig
Releases · SMC17/tokenizers-zig
Release list
v0.26.0 — property fuzzer + canonical Encoding parity
Property-based invariant fuzzer caught a real BPE+SP-marker offset bug. 189 tests pass + 600 random texts cross all three real fixtures clean.
Property fuzz (tests/test_fuzz_invariants.zig)
For each real fixture (BERT, T5, TinyLlama), generate 200 random ASCII texts and verify:
- Non-empty input → ≥1 token
- All offsets within [0, text.len]
- Offsets monotonic non-decreasing
- attention_mask / type_ids / offsets / tokens / special_tokens_mask all parallel to ids
- Decode does not error
Bug caught + fixed
TinyLlama (BPE+SP marker) offset remap used chunk_text.len as fallback for all-SYNTHETIC tokens (e.g. double-space ▁▁). Violated monotonicity. Fix: running_end cursor through encode loop. Caught on text iHuL?OXlo,kxi0nl7a VfXytl — two adjacent spaces.
v0.25 + v0.24 highlights
- Full canonical HF
Encodingstruct + one-shotencodeFull/encodePairFull encodeWithOffsetsbench: ~7.9 μs/encode on BERT-base-uncased (vs ~14.6 μs for full template-wrapped encode)
Full changelog: CHANGELOG.md