[Bug] session-query-sqlite: CJK substrings are not searchable, and one unreadable Session fails every search

[xiangxiaoyong]

Version: 0.2.0-rc.2 (@deepseek-ai/dsh-session-query-sqlite)

There are two issues in the official history search index.

1. CJK text is not searchable by word. The FTS5 index uses the plain unicode61 tokenizer, which keeps a run of Han characters as a single token. Searching 刑事 does not find a message containing 请分析这个刑事案件. Any Chinese or Japanese query that is not the entire run of characters misses.

• Our local workaround: insert a reserved noncharacter (U+FDD2) between Han code points, declare it as an extra unicode61 separator, and prepare the query the same way. Phrase order is kept. Snippets strip the separator, and stored lengths use the original text. This bumps the derived index schema version, so the index is rebuilt once.
• A trigram tokenizer, or ICU segmentation, would be the cleaner upstream fix.

2. One unreadable Session fails every search. When a stored Session is refused by the reader, for example a 0.1.5 log rejected with SessionFormatUnsupportedError, every search fails with SESSION_QUERY_PERSISTENCE_FAILED. So a single retired Session disables search for the whole host.

• Our patch leaves exactly that error class out of the index. Every other read error still fails the search.

Ask. Index CJK text so substrings match, and skip (and report) Sessions the reader cannot read, instead of failing every query.

[QIANLING-0831]

For the CJK substring case, dsh-memory-plus / dsh-session-query-sqlite-cjk provides a backend you can try. It keeps parallel unicode61 and trigram tables, routes CJK queries to trigram, and uses escaped LIKE for 1–2-code-point queries. That fallback covers your two-character 刑事 example; supplementary Han characters are covered too.

CJK host verification records DSH 0.2.0-rc.2 Web loading, persistent history, cold restart and sidebar search. v0.2.2 now ships an installable bundle with checksums, so a full checkout is no longer required.

For the unreadable-session case, the indexing pipeline still needs a separate skip/report policy. Keeping that acceptance test separate from CJK recall should make either backend easier to validate.