Skip to content

Releases

NTokenizer v0.3-polished

Choose a tag to compare

@Neuratal Neuratal released this 16 Nov 10:02

NTokenizer v0.3-polished (v0.3.0)

NTokenizer is an independently developed, high-performance multilingual byte-level BPE tokenizer written in Rust by Neuratal. Founder and creator: Rüzgar Erdem.

What this is

  • Tokenizer, not a language model. Byte-level BPE; byte-exact encode/decode for arbitrary bytes.
  • Vocabulary: 262,144 tokens (256 byte base + 261,881 learned merges + 7 special tokens).
  • Model of record: ntokenizer-0.3-256k.ntok (SHA-256 1f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9); download from Hugging Face: https://huggingface.co/Neuratal/ntokenizer

Development-benchmark results (544 files, development/validation set — not a blind test)

Metric v0.3-polished
Overall byte-weighted B/T 5.5542
Equal-category B/T 5.1056
Byte-exact decode round-trip 544/544
Vocabulary utilized 93.59%

Historical v0.3 report recorded 37,397,108 tokens / 5.5543; the polished tree encodes the same benchmark to 37,397,319 tokens / 5.5542 (+211 tokens, +0.00056%, disclosed; see README).

Correctness and testing

  • 136/136 Rust tests pass.
  • Adversarial Unicode regression suite: Turkish dotted/dotless I, Devanagari vowel signs, Arabic harakat, Thai vowel signs, combining diacritics, emoji ZWJ sequences, skin tones, variation selectors, flags, mixed scripts — all byte-exact.
  • Fuzzing: ~924K roundtrip runs and ~83K model-loading runs, zero crashes (ASAN-instrumented build remains blocked by a pre-existing linker issue; not claimed).

Licenses

  • Source code: MIT OR Apache-2.0
  • Model artifacts: CC BY-SA 4.0 (ShareAlike inherited from CC BY-SA training text). See NOTICE.

Known limitations

  • Development benchmark, not a pristine blind test set.
  • CJK tokenization imperfect (CJK chunking candidates rejected).
  • HF tokenizer.json export: byte-exact both directions, 3/16 token-ID parity (documented).
  • No downstream 50M LM evaluation yet.