Feature proposal: configurable FTS5 tokenizer for session search (Chinese support)

[corntrace]

Feature proposal: configurable FTS5 tokenizer for session search (Chinese support)

中文摘要：session-query-sqlite 的全文索引目前硬编码 tokenize = 'unicode61'（SQLite 默认分词器）。unicode61 不做中文分词，一段连续中文（含中英混排）会被当成一个超长 token，导致中文 session 搜索几乎无法命中。建议给该包增加 tokenizer / tokenizerExtension 两个配置项，允许加载第三方分词扩展（如 wangfenjin/simple，SQLite FTS5 中文+拼音分词，MIT/GPLv3 双许可），并顺手把索引 schema 版本升到 9 以在切换分词器时自动重建索引。

Problem

With the default unicode61 tokenizer, a document like 查询OpenRouter免费视觉模型 becomes one index token (查询openrouter免费视觉模型). Querying 视觉模型 or OpenRouter matches nothing:

query	unicode61 (current)	simple tokenizer
视觉模型	0 hits	✅
免费	0 hits	✅
OpenRouter (inside CJK run)	0 hits	✅

Proposed change

@deepseek-ai/dsh-session-query-sqlite gains two optional config keys (default behavior unchanged):

- id: session-query-sqlite
  config:
    path: ':memory:'
    openAt: first-search
    tokenizer: simple
    tokenizerExtension: /path/to/libsimple.so

Implementation notes:

• tokenizer is restricted to [A-Za-z0-9_]+ (validated in the schemastery schema, matching how openAt is validated) so configuration can never interpolate SQL.
• When tokenizerExtension is set, the engine opens node:sqlite with allowExtension, calls enableLoadExtension + loadExtension, then creates the FTS5 tables with the configured tokenizer; otherwise everything stays exactly as today (unicode61, no extension loading).
• SESSION_QUERY_SQLITE_SCHEMA_VERSION bumps 8 → 9 so existing persistent derived indexes rebuild with the configured tokenizer.
• A non-default tokenizer without tokenizerExtension is rejected at config resolution.

Validation

• Package tests: 72/72 pass, including new tests for tokenizer config validation and a skip-gated test that loads a real simple extension and searches Chinese phrases (skipped in CI when the extension is absent).
• pnpm run typecheck, oxlint, verify-config-catalog all pass; docs/config-catalog.md regenerated.

Ready-to-merge branch

The full change is committed on feat/session-query-tokenizer in my fork: https://github.com/corntrace/deepseek-harness/tree/feat/session-query-tokenizer

I understand external PRs are not accepted at the moment (per CONTRIBUTING.md) — posting this as an Idea per the contributing guidance. If the team is interested, the branch is one click away from a PR; happy to adjust scope, add i18n docs, or split the change.

[QIANLING-0831]

数据点 + 两条实现约束，来自一个独立实现（trigram 路线，不是扩展加载路线）

I maintain a separate fork implementing CJK search on the same seam, and I reproduced the core of this proposal before reading it. Two things that may help scope the change:

1. Independent confirmation of the premise. Measured on node:sqlite + FTS5 (same engine DSH uses) against a document containing 索引优化减少Token消耗的句子:

query	unicode61	trigram table
Token消耗 (mixed CJK+ASCII)	0 hits	✅
索引优化	0 hits	✅
中文分词	0 hits	✅
the full sentence	✅ (the only way)	✅
消耗 (2 chars)	0 hits	❌ still 0
优 (1 char)	0 hits	❌ still 0

Your 查询OpenRouter免费视觉模型 example matches this exactly — the ASCII inside a CJK run is swallowed into the same token.

2. Constraint worth putting in the proposal: the tokenizer choice alone does not fix short queries.

• 1–2 character CJK queries. A trigram index contains no 1- or 2-character gram, so MATCH on a trigram table can never hit them — measured above, the last two rows. These are the most common Chinese query shape. Any tokenizer-based fix needs a defined answer for them; in my fork they route to a LIKE '%term%' ESCAPE '\' scan. If the simple extension segments properly, it presumably solves this without the scan — worth stating explicitly in the proposal, because a reviewer will otherwise assume trigram "done" means CJK "done".
• Index size. The parallel trigram table measured roughly 2–3× the source text in my implementation. Since your change rebuilds the derived index on schema bump anyway, it may be worth mentioning the storage delta next to the recall delta, so adopters are not surprised.

3. A no-extension alternative worth having in the same discussion. My fork avoids loadExtension entirely (no native artifact to ship, works from plain JS + built-ins): keep the upstream unicode61 table untouched for ASCII, add a trigram table written in parallel, route by whether the query contains CJK, and fall back to escaped LIKE for 1–2 character queries. Pure-ASCII behaviour stays byte-identical to upstream, which matters for a service the whole ecosystem sits on. It is not better than a real segmenter for precision — a segmenter should win there — but it is deployable without a .so/.dll, which for a plugin maintained by a community fork is a real consideration.

Source with the measurements and the test suite: https://github.com/QIANLING-0831/dsh-memory-plus (the dsh-session-query-sqlite-cjk package; MIT). Benchmarks and trade-offs are in that package's README — I have deliberately not repeated a recall/precision table here because mine is a different scheme and I have not run a head-to-head against simple.

Also relevant: @xzy-jason's jieba-based approach in #1456 reports 94% precision vs 45% for trigram+LIKE on 1,000 Chinese entries. If the team is weighing a tokenizer seam, that thread's data plus yours covers both the extension route and the pure-JS route.

---

中文摘要：我维护另一个在同一条接缝上实现中文检索的 fork（trigram 路线，不加载原生扩展），这里补两点：① 用 node:sqlite + FTS5 独立复现了你的结论（Token消耗 / 索引优化 在 unicode61 下 0 命中）；② 提醒一个容易漏的点——只换 tokenizer 并不自动解决 1–2 字中文查询，trigram 索引里根本没有 1–2 字 gram，我那两条短查询实测仍是 0 命中，需要单独定义回退（我这边走 LIKE ... ESCAPE）；另外并存一张 trigram 表体积约为原文 2–3 倍，建议和召回收益一起写进提案。同类方案还有 #1456 的 jieba 双索引（自报精确率 94%）。这几条合起来，官方权衡"扩展加载 vs 纯 JS"时应该有足够信息。