Repository navigation
NTokenizer v0.3-polished (v0.3.0)
NTokenizer is an independently developed, high-performance multilingual byte-level BPE tokenizer written in Rust by Neuratal. Founder and creator: Rüzgar Erdem.
What this is
- Tokenizer, not a language model. Byte-level BPE; byte-exact encode/decode for arbitrary bytes.
- Vocabulary: 262,144 tokens (256 byte base + 261,881 learned merges + 7 special tokens).
- Model of record:
ntokenizer-0.3-256k.ntok(SHA-2561f0d4ad7c7fb609001b2af4d9741ec5b8a5d7686fa3b92eb8eefd057fea936d9); download from Hugging Face: https://huggingface.co/Neuratal/ntokenizer
Development-benchmark results (544 files, development/validation set — not a blind test)
| Metric | v0.3-polished |
|---|---|
| Overall byte-weighted B/T | 5.5542 |
| Equal-category B/T | 5.1056 |
| Byte-exact decode round-trip | 544/544 |
| Vocabulary utilized | 93.59% |
Historical v0.3 report recorded 37,397,108 tokens / 5.5543; the polished tree encodes the same benchmark to 37,397,319 tokens / 5.5542 (+211 tokens, +0.00056%, disclosed; see README).
Correctness and testing
- 136/136 Rust tests pass.
- Adversarial Unicode regression suite: Turkish dotted/dotless I, Devanagari vowel signs, Arabic harakat, Thai vowel signs, combining diacritics, emoji ZWJ sequences, skin tones, variation selectors, flags, mixed scripts — all byte-exact.
- Fuzzing: ~924K roundtrip runs and ~83K model-loading runs, zero crashes (ASAN-instrumented build remains blocked by a pre-existing linker issue; not claimed).
Licenses
- Source code: MIT OR Apache-2.0
- Model artifacts: CC BY-SA 4.0 (ShareAlike inherited from CC BY-SA training text). See NOTICE.
Known limitations
- Development benchmark, not a pristine blind test set.
- CJK tokenization imperfect (CJK chunking candidates rejected).
- HF
tokenizer.jsonexport: byte-exact both directions, 3/16 token-ID parity (documented). - No downstream 50M LM evaluation yet.