fix(capture): skip oversized SQLite rows instead of aborting source#88
Merged
Merged
Conversation
Providers like the locally-installed Z.ai Kilo client can carry single
messages > 100 MB (e.g. a multi-file diff summary) that legitimately
exceed ctx's per-value import cap. Previously one oversized row aborted
the entire source import, which in turn made ctx search --refresh
strict refuse to refresh the index for any provider sharing the
invocation.
Skip oversized rows instead:
* Pre-scan oversized row ids on a separate unbounded read connection
(the bounded connection enforces SQLITE_LIMIT_LENGTH even when
computing length(), so the cap must be off to enumerate offenders).
* Exclude those ids at the SQL layer via where id not in (...) —
sqlite3_step() materializes every column of the current row, so
filtering after
ows.next() is too late.
* Keep a per-row try/catch on the data column as a safety net for
edge cases the pre-scan might miss (e.g. multi-byte UTF-8 inserted
between the two connections).
* Surface the skip count via the import summary's failure list so
ctx import and ctx search --refresh both report what was dropped.
The pre-scan runs after schema validation so 'missing required column'
errors for future/incompatible provider schemas still surface first.
robbe1912
force-pushed
the
fix/search-refresh-oversized-skip
branch
from
July 7, 2026 12:26
837db98 to
64b82f6
Compare
This was referenced Jul 7, 2026
robbe1912
force-pushed
the
fix/search-refresh-oversized-skip
branch
from
July 7, 2026 13:06
353418e to
6dae8ec
Compare
Match repo style (ctx-history-capture crate has near-zero inline comments across all provider adapters). Move design rationale to PR description; keep only load-bearing items in code. Also correct provider attribution in the one retained comment: Kilo Code is the VSCode extension, not a z.ai product. z.ai is the coding-plan/model provider; the local sqlite store ctx reads belongs to the Kilo Code extension itself.
robbe1912
force-pushed
the
fix/search-refresh-oversized-skip
branch
from
July 7, 2026 13:06
6dae8ec to
cdf8045
Compare
robbe1912
marked this pull request as ready for review
July 7, 2026 13:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Provider SQLite databases can contain
datavalues larger than ctx's 16 MiB per-value import cap. Before this PR, materializing one oversized value could abort the entire source import withSQLITE_TOOBIG, leaving otherwise readable rows unindexed.This keeps the cap and makes the OpenCode-family SQLite adapter omit oversized values at the SQL layer instead of aborting the source:
data;SQLITE_TOOBIGsafety net for rows that slip past the prescan;skipped/skipped_events, not failures;No synthetic text, fake previews, truncated diffs, or summary events are generated. Oversized payload content is omitted.
Affected scope: OpenCode-family SQLite rows currently read by
normalize_opencode_sqlite, including OpenCode/Kilo dialects. JSONL adapters are unchanged.Validation
cargo fmt --checkcargo test -p ctx-history-capture --lib tests::native_sqlitecargo test -p ctx-history-captureRegression coverage includes mixed readable + oversized rows, all-oversized rows, and legacy
messagetable rows.