feat: support new Gemini CLI JSONL format and filter metadata#404
Conversation
…ion clobbering The JSONL parser now mirrors gemini-cli's own loadConversationRecord semantics instead of approximating them: - Handle $rewindTo records — drop the target message and everything after it (previously rewound-away turns stayed in the index forever). - Treat $set.messages as a checkpoint that replaces all prior messages, matching the upstream clear-and-rebuild, instead of merging. - Replace same-id message snapshots wholesale (tool-call status updates) rather than shallow-merging, and key replay off a Map to drop the O(n²) findIndex scan per record. Sync-layer fixes for the format transition: - Skip a legacy session-*.json when its migrated .jsonl sibling exists. Resuming a legacy session in gemini-cli ≥0.39 leaves both files on disk with the same sessionId; indexing both made them clobber each other's session row via UNIQUE(session_uuid) on every scan. - classifySync now also compares the stored tail uuid for gemini: a rewind followed by enough new turns keeps parsed.length >= total with the head unchanged, which the head+count heuristic misread as append, stranding rewound rows with colliding seq values. - An index-version change in the stored mtime suffix forces the rewrite path; append-mode INSERT … DO NOTHING would re-visit every file after a bump and update nothing. Bump gemini to v2 so sessions indexed before <session_context> stripping re-derive their content. Verified against real session files from gemini-cli 0.38.2 (legacy .json), 0.49.0 (JSONL), and a 0.49.0 --resume migration of a 0.38.2 session: three syncAll rounds converge with no file_path flip-flop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for this — the format support is real and needed (verified against gemini-cli source: JSONL landed upstream in 0.39.0, and Parser (vs upstream replay semantics)
Sync layer (format-transition hazards)
Also confirmed your All new tests fail on the pre-fix code and pass after; core suite is green (404 passed). One known limitation left as-is (pre-existing, applies to the legacy format too): an in-progress same-id snapshot indexed mid-turn by the live watcher won't have its content updated by later snapshots until a rewind/truncation or an explicit "Refresh from source" — content-comparing uuid-matched rows is deliberately avoided to protect Security Scan purge state (see the pinned test in syncer.test.ts). |
This PR adds support for the new Gemini CLI JSONL session transcript format and filters metadata blocks to prevent FTS/title indexing pollution.
Changes
packages/core/src/parsers/gemini.tsthat handles streaming JSON lines (supporting direct message entries and the$setoperator).<session_context>blocks in user inputs to prevent metadata pollution.packages/core/src/sync/source-paths.tsto include.jsonlpatterns in watch lists and fix nestedtmp/tmpworkspace folder resolution.