v0.8.2 — every script priced at its real token rate
v0.8.0 moved CJK onto a real token rate. It left every other non-Latin script on the English one, which is the same defect with a smaller multiplier — a Russian, Greek, Arabic, Hebrew or Hindi chunk still ran about twice its intended size, and a Thai chunk three to four times.
Chunk size is a budget in tokens, so mispricing a script means its readers pay that multiple in context on every retrieved hit — permanently, for writing in their own language.
Same sentence, estimated tokens
| characters | before | after | |
|---|---|---|---|
| English | 50 | 12 | 12 |
| Korean | 29 | 25 | 25 |
| Japanese | 31 | 31 | 31 |
| Russian | 55 | 13 | 26 |
| Thai | 56 | 14 | 56 |
| Hindi | 49 | 12 | 22 |
| Arabic | 47 | 11 | 21 |
| Greek | 46 | 11 | 21 |
Three rates
- dense and unspaced (Han, kana, Hangul, Thai, Lao, Khmer, Myanmar) — about one token per character
- other non-ASCII (Cyrillic, Greek, Arabic, Hebrew, Indic, accented Latin) — about two characters per token
- ASCII — four characters per token
Hardcoded code-point ranges are gone in favour of Script_Extensions, so a character like the prolonged-sound mark ー (which is Script=Common) is no longer missed.
Upgrading: chunk boundaries changed, so an existing index invalidates and rebuilds itself on first open. Nothing under docs/wiki is touched. Measured on a 245-page wiki: 0.70s.
590 tests, tsc --noEmit clean.