Skip to content

v0.8.1 — per-turn injection works in every writing system

Choose a tag to compare

@suwonleee suwonleee released this 25 Jul 14:04

v0.8.0 fixed search, dates, links and chunking — but its verification was CJK-and-Latin shaped, and it never checked the per-turn injection path. Measured properly, that path worked in three of nine languages.

What was broken

Injection runs off terms pulled out of your prompt, and the two patterns doing the pulling recognised ASCII and CJK only.

  • Cyrillic, Thai, Devanagari, Arabic, Greek, Hebrew produced no terms at all. Not degraded — the feature never fired once.
  • Japanese and Chinese failed the other way: with no spaces to break on, an entire clause became one term, and an entire clause is a literal substring no page contains.
  • Vietnamese produced one term: the ASCII pattern cannot start on a letter with a diacritic, so nearly every syllable was dropped. The same fragmentation quietly cost German, French, Spanish, Portuguese, Turkish and Polish their accented words (überhaupt arrived as berhaupt).

Measured, before and after

Fair fixtures — the page and the prompt share the same content words in each language:

en ko ja zh ru th hi ar vi
before
after

All eight unrelated control prompts stay silent, before and after — recall improved without spending precision.

How

  • a word pattern for every script, with \p{M} keeping combining marks attached
  • runs in scripts written without spaces (Han, Kana, Thai, Lao, Khmer, Myanmar) sliced into word-sized windows. Script_Extensions, not Script — the prolonged-sound mark ending マイグレーション is Script=Common and split the run down the middle
  • floors kept honest: 4 characters for Latin, 3 for a dense script, so the the/and/for class still stays out without a stopword list

588 tests, tsc --noEmit clean. No index migration — this release changes only how a prompt is read.