Skip to content

fix(capture): skip oversized SQLite rows instead of aborting source#88

Merged
luca-ctx merged 4 commits into
ctxrs:mainfrom
robbe1912:fix/search-refresh-oversized-skip
Jul 7, 2026
Merged

fix(capture): skip oversized SQLite rows instead of aborting source#88
luca-ctx merged 4 commits into
ctxrs:mainfrom
robbe1912:fix/search-refresh-oversized-skip

Conversation

@robbe1912

@robbe1912 robbe1912 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Provider SQLite databases can contain data values larger than ctx's 16 MiB per-value import cap. Before this PR, materializing one oversized value could abort the entire source import with SQLITE_TOOBIG, leaving otherwise readable rows unindexed.

This keeps the cap and makes the OpenCode-family SQLite adapter omit oversized values at the SQL layer instead of aborting the source:

  • pre-scan relevant tables with a read-only, query-only connection that returns row ids only;
  • exclude oversized ids from the bounded import query before SQLite materializes data;
  • keep a per-row SQLITE_TOOBIG safety net for rows that slip past the prescan;
  • count omitted rows as skipped / skipped_events, not failures;
  • continue importing every other readable row.

No synthetic text, fake previews, truncated diffs, or summary events are generated. Oversized payload content is omitted.

Affected scope: OpenCode-family SQLite rows currently read by normalize_opencode_sqlite, including OpenCode/Kilo dialects. JSONL adapters are unchanged.

Validation

  • cargo fmt --check
  • cargo test -p ctx-history-capture --lib tests::native_sqlite
  • cargo test -p ctx-history-capture

Regression coverage includes mixed readable + oversized rows, all-oversized rows, and legacy message table rows.

Providers like the locally-installed Z.ai Kilo client can carry single
messages > 100 MB (e.g. a multi-file diff summary) that legitimately
exceed ctx's per-value import cap. Previously one oversized row aborted
the entire source import, which in turn made ctx search --refresh
strict refuse to refresh the index for any provider sharing the
invocation.

Skip oversized rows instead:

  * Pre-scan oversized row ids on a separate unbounded read connection
    (the bounded connection enforces SQLITE_LIMIT_LENGTH even when
    computing length(), so the cap must be off to enumerate offenders).
  * Exclude those ids at the SQL layer via where id not in (...) —
    sqlite3_step() materializes every column of the current row, so
    filtering after 
ows.next() is too late.
  * Keep a per-row try/catch on the data column as a safety net for
    edge cases the pre-scan might miss (e.g. multi-byte UTF-8 inserted
    between the two connections).
  * Surface the skip count via the import summary's failure list so
    ctx import and ctx search --refresh both report what was dropped.

The pre-scan runs after schema validation so 'missing required column'
errors for future/incompatible provider schemas still surface first.
Match repo style (ctx-history-capture crate has near-zero inline
comments across all provider adapters). Move design rationale to PR
description; keep only load-bearing items in code.

Also correct provider attribution in the one retained comment:
Kilo Code is the VSCode extension, not a z.ai product. z.ai is the
coding-plan/model provider; the local sqlite store ctx reads belongs
to the Kilo Code extension itself.
@robbe1912
robbe1912 force-pushed the fix/search-refresh-oversized-skip branch from 6dae8ec to cdf8045 Compare July 7, 2026 13:06
@robbe1912
robbe1912 marked this pull request as ready for review July 7, 2026 13:35
@luca-ctx
luca-ctx merged commit 6fd71de into ctxrs:main Jul 7, 2026
1 check failed
@robbe1912
robbe1912 deleted the fix/search-refresh-oversized-skip branch July 7, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants