Skip to content

feat: support new Gemini CLI JSONL format and filter metadata#404

Merged
graydawnc merged 2 commits into
paperboytm:mainfrom
bet4it:feat/gemini-jsonl
Jul 7, 2026
Merged

feat: support new Gemini CLI JSONL format and filter metadata#404
graydawnc merged 2 commits into
paperboytm:mainfrom
bet4it:feat/gemini-jsonl

Conversation

@bet4it

@bet4it bet4it commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

This PR adds support for the new Gemini CLI JSONL session transcript format and filters metadata blocks to prevent FTS/title indexing pollution.

Changes

  1. JSONL Format Parser: Added a JSONL parser in packages/core/src/parsers/gemini.ts that handles streaming JSON lines (supporting direct message entries and the $set operator).
  2. Metadata Filtering: Implemented XML-style tag filtering for <session_context> blocks in user inputs to prevent metadata pollution.
  3. Session Paths: Updated packages/core/src/sync/source-paths.ts to include .jsonl patterns in watch lists and fix nested tmp/tmp workspace folder resolution.

bet4it and others added 2 commits July 5, 2026 22:30
…ion clobbering

The JSONL parser now mirrors gemini-cli's own loadConversationRecord
semantics instead of approximating them:

- Handle $rewindTo records — drop the target message and everything
  after it (previously rewound-away turns stayed in the index forever).
- Treat $set.messages as a checkpoint that replaces all prior messages,
  matching the upstream clear-and-rebuild, instead of merging.
- Replace same-id message snapshots wholesale (tool-call status
  updates) rather than shallow-merging, and key replay off a Map to
  drop the O(n²) findIndex scan per record.

Sync-layer fixes for the format transition:

- Skip a legacy session-*.json when its migrated .jsonl sibling exists.
  Resuming a legacy session in gemini-cli ≥0.39 leaves both files on
  disk with the same sessionId; indexing both made them clobber each
  other's session row via UNIQUE(session_uuid) on every scan.
- classifySync now also compares the stored tail uuid for gemini:
  a rewind followed by enough new turns keeps parsed.length >= total
  with the head unchanged, which the head+count heuristic misread as
  append, stranding rewound rows with colliding seq values.
- An index-version change in the stored mtime suffix forces the
  rewrite path; append-mode INSERT … DO NOTHING would re-visit every
  file after a bump and update nothing. Bump gemini to v2 so sessions
  indexed before <session_context> stripping re-derive their content.

Verified against real session files from gemini-cli 0.38.2 (legacy
.json), 0.49.0 (JSONL), and a 0.49.0 --resume migration of a 0.38.2
session: three syncAll rounds converge with no file_path flip-flop.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@graydawnc

Copy link
Copy Markdown
Collaborator

Thanks for this — the format support is real and needed (verified against gemini-cli source: JSONL landed upstream in 0.39.0, and <session_context> injection is exactly as you described). I reproduced the full matrix locally with real session files from 0.38.2 (legacy .json), 0.49.0 (JSONL), and a 0.49.0 --resume migration of a 0.38.2 session, and found a few gaps against upstream's own loader semantics (chatRecordingService.tsloadConversationRecord). I've pushed a commit (76876e3) on top of your branch that addresses them:

Parser (vs upstream replay semantics)

  • $rewindTo records were ignored — rewound-away messages stayed in the index forever. Now dropped (target + everything after), matching upstream.
  • $set.messages is a checkpoint upstream (messagesMap.clear() + rebuild); the merge approach kept messages that a checkpoint had removed. Now clear-and-rebuild.
  • Same-id snapshots (tool-call status updates) are full replacements upstream; shallow-merge could keep stale fields a later snapshot legitimately dropped. Replay is now keyed off a Map, which also removes the O(n²) findIndex per record.
  • <session_context> stripping now applies to user messages only, so assistant output quoting the tag isn't mangled.

Sync layer (format-transition hazards)

  • Resuming a legacy session in ≥0.39 migrates it to a sibling .jsonl and leaves the old .json in place — same sessionId in both. Indexing both made them clobber each other's row via UNIQUE(session_uuid) on every scan (reproduced: syncAll never converged, file_path flip-flopped each round). A .json with a .jsonl sibling is now skipped.
  • A rewind followed by enough new turns keeps the replayed count ≥ the stored row count with the head unchanged, which the append/rewrite heuristic misread as append — rewound rows survived with colliding seq values. classifySync now also checks the stored tail uuid for gemini.
  • Bumped the gemini index version to v2 and made a version change force the rewrite path (append-mode INSERT … DO NOTHING re-visits files after a bump but updates nothing), so sessions indexed before the stripping re-derive their content.

Also confirmed your tmp/tmp fix addresses a real bug — the workspace dir under ~/.gemini/tmp/ is named after the cwd's basename, so running gemini inside any directory named tmp creates ~/.gemini/tmp/tmp/, and the old root normalization then re-nested the root and made every gemini session invisible. Nice catch; I added a regression test for it.

All new tests fail on the pre-fix code and pass after; core suite is green (404 passed). One known limitation left as-is (pre-existing, applies to the legacy format too): an in-progress same-id snapshot indexed mid-turn by the live watcher won't have its content updated by later snapshots until a rewind/truncation or an explicit "Refresh from source" — content-comparing uuid-matched rows is deliberately avoided to protect Security Scan purge state (see the pinned test in syncer.test.ts).

@graydawnc
graydawnc added this pull request to the merge queue Jul 7, 2026
Merged via the queue into paperboytm:main with commit 4b34c93 Jul 7, 2026
3 checks passed
@bet4it
bet4it deleted the feat/gemini-jsonl branch July 7, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants