Skip to content

Add multilingual scriptgram search#101

Merged
luca-ctx merged 1 commit into
mainfrom
ctx/multilingual-scriptgram-search
Jul 8, 2026
Merged

Add multilingual scriptgram search#101
luca-ctx merged 1 commit into
mainfrom
ctx/multilingual-scriptgram-search

Conversation

@luca-ctx

@luca-ctx luca-ctx commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add scriptgram FTS sidecar tables for scripts that default SQLite FTS tokenizes poorly
  • merge scriptgram hits with normal record/event FTS only for covered non-Latin queries
  • add v45 migration/backfill plus multilingual regression coverage

Impact

English/code-only text and queries stay on the existing FTS path and do not populate the sidecar. Multilingual rows get additional sidecar tokens for Japanese, Chinese, Korean, Thai/Lao/Khmer/Myanmar, and major Indic scripts.

Validation

  • cargo test -p ctx-history-store
  • cargo test -p ctx-history-search
  • cargo test --workspace
  • dogfooded real ctx import/search against ctx-history-jsonl-v1 fixture for 認証, OAuth 認証, 登录态, 오류, and SearchResultScope

@luca-ctx
luca-ctx merged commit 05f1f8c into main Jul 8, 2026
1 check was pending
@luca-ctx
luca-ctx deleted the ctx/multilingual-scriptgram-search branch July 8, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant