Skip to content

Releases: SMC17/tokenizers-zig

v0.26.0 — property fuzzer + canonical Encoding parity

Choose a tag to compare

@SMC17 SMC17 released this 28 May 01:45

Property-based invariant fuzzer caught a real BPE+SP-marker offset bug. 189 tests pass + 600 random texts cross all three real fixtures clean.

Property fuzz (tests/test_fuzz_invariants.zig)

For each real fixture (BERT, T5, TinyLlama), generate 200 random ASCII texts and verify:

  1. Non-empty input → ≥1 token
  2. All offsets within [0, text.len]
  3. Offsets monotonic non-decreasing
  4. attention_mask / type_ids / offsets / tokens / special_tokens_mask all parallel to ids
  5. Decode does not error

Bug caught + fixed

TinyLlama (BPE+SP marker) offset remap used chunk_text.len as fallback for all-SYNTHETIC tokens (e.g. double-space ▁▁). Violated monotonicity. Fix: running_end cursor through encode loop. Caught on text iHuL?OXlo,kxi0nl7a VfXytl — two adjacent spaces.

v0.25 + v0.24 highlights

  • Full canonical HF Encoding struct + one-shot encodeFull / encodePairFull
  • encodeWithOffsets bench: ~7.9 μs/encode on BERT-base-uncased (vs ~14.6 μs for full template-wrapped encode)

Full changelog: CHANGELOG.md