`jcode session migrate`: session storage migration & v2 sidecar fast-path for large sessions

[quangdang46]

jcode session migrate: session storage migration & v2 sidecar fast-path for large sessions

Summary

Add a session-storage migration command (jcode session migrate [path] [--dry-run]) and a v2 sidecar format that pairs the existing JSONL log with an indexed SQLite sidecar so resume and /tree open large sessions in <100 ms. Mirrors pi migrate and pi_agent_rust's "Session Store V2 Sidecar".

Reference: pi_agent_rust Session Indexing and Session Store V2 Sidecar (Large Session Fast-Path).

Why

• jcode sessions can balloon past tens of thousands of entries (especially in self-dev). Resume time on those is currently O(N) read + parse.
• The session-tree issue introduces parent_id; rebuilding the tree on every open is wasteful without an index.
• A separate migration command lets users run it in CI before upgrading.

Current state in jcode

• src/session_active_pids.rs and src/storage/ exist; no v2 sidecar, no migration command.
• Resume picker (src/tui/session_picker/loading.rs) parses each JSONL on demand, which gets slow on large sessions.

Implementation checklist

1. v2 sidecar schema

• Per JSONL session file, an adjacent .sqlite3 sidecar with tables:
  • entries(id PK, parent_id, ts, kind, branch_leaf_marker, byte_offset, byte_len)
  • branches(leaf_id PK, name, labeled_at)
  • metadata(key PK, value)
• Sidecar is regenerable from the JSONL at any time. Mismatched sidecar → treat as corrupt and rebuild.

2. Read path

• On open:
  1. mmap the JSONL.
  2. If sidecar exists and is up to date (mtime + size match metadata), use it for index queries.
  3. Otherwise, rebuild sidecar in the background while serving reads from the JSONL.

3. Write path

• Every appended entry updates the sidecar in the same transaction the JSONL is fsync'd in.
• Durability mode (see separate consideration): strict (fsync every write), balanced (fsync every N entries or T seconds), throughput (fsync on flush only). Pi_agent_rust ships these as --session-durability.

4. CLI subcommand

• jcode session migrate [path] [--dry-run] [--force]
  • Default path: ~/.jcode/sessions/ recursively.
  • --dry-run: print what would be migrated; no writes.
  • --force: rebuild sidecars even when up to date.
• Subcommand also exposes jcode session verify <path> to check sidecar/JSONL coherence.

5. Resume picker speedup

• When a sidecar is present, the picker reads metadata (name, last activity, message count, branch count) from the sidecar without parsing the JSONL.

6. Doctor integration

• jcode doctor --only sessions reports number of sessions, total size, sidecar coverage, last migrate time.

Testing

Unit

• Sidecar build is deterministic for a given JSONL.
• Corruption recovery: zero-out random bytes in the sidecar; on open, rebuild succeeds and JSONL data is intact.
• Durability mode crash test (using a fault-injecting writer): strict mode survives a kill -9 between writes with no entry loss.

Bench

• Resume picker open time for a 100,000-entry session: <100 ms with sidecar, document baseline without.

Manual

• Convert a real long session and confirm /tree opens fast.

Acceptance criteria

• jcode session migrate is idempotent and lossless.
• Sidecars are auto-rebuilt on detection of mismatch.
• --dry-run previews accurately.
• Durability modes are documented and selectable per-session.
• Resume picker shows the speed-up on the bench above.

References

• pi_agent_rust: Session Indexing, Session Store V2 Sidecar, Subcommand pi migrate.
• pi: Session format.

---

---

Tracking: physically split into sub-issues

This issue is now an umbrella. Concrete work lives in:

• [#17a] Session storage v2 sidecar: SQLite-backed session index with WAL, lock, stale reindex #33 — [#17a] SQLite session index sidecar (WAL + lock + stale reindex)
• [#17b] Session persistence: crash-resilient temp-file + atomic-persist save path #34 — [#17b] Crash-resilient session save (temp file + atomic persist)
• [#17c] --session-durability strict|balanced|throughput flag #35 — [#17c] --session-durability strict|balanced|throughput flag
• [#17d] Session rollback ledger: periodic checkpoints + jcode session rollback #36 — [#17d] Rollback ledger + periodic checkpoints + jcode session rollback
• [#17e] jcode session migrate subcommand: idempotent v1 → tree-aware schema upgrade #37 — [#17e] jcode session migrate subcommand (idempotent v1 → tree schema)

Land in order. Close this issue when all five sub-issues are closed.

---

Implementation notes addendum (Devin gap-analysis pass, 2026-05-21)

This issue should be split — recommend 5 sub-issues

The original body says "session migration & v2 sidecar fast-path" but pi_agent_rust treats this as five separable deliverables. Land them in order so each one is independently shippable and testable:

1. #17a — v2 sidecar SQLite index (fast resume/list)

   • SQLite file at ~/.jcode/agent/sessions.idx.db with WAL + a *.lock file.
   • Indexes: (cwd, last_modified), (session_id), (tag).
   • Stale-detection: if any indexed JSONL mtime newer than its row, mark stale; reindex affected rows on first read.

2. #17b — Crash-resilient save path (temp file + atomic persist)

   • Replace appendFileSync-style writes with write_temp_then_rename. On crash mid-write, the previous version of the file is intact.
   • Implement in src/session.rs.

3. #17c — --session-durability strict|balanced|throughput flag

   • strict: fsync after every entry (slow, safest)
   • balanced (default): fsync every N entries or T ms
   • throughput: rely on OS page cache

4. #17d — Rollback ledger + checkpoints + segmented log

   • Periodically snapshot the current head, store a small ledger of (timestamp, branch_leaf, checkpoint_path).
   • jcode session rollback <session-id> [--to-checkpoint <ts>].

5. #17e — jcode session migrate subcommand

   • One-shot v1→v3 migration plus future-proofing for v3→v4.
   • --dry-run, --all, --session <id>.
   • Idempotent on already-migrated files.

Verified jcode code paths

• Existing session persistence: src/session.rs, src/session/.
• Picker that benefits from a fast index: src/tui/session_picker.rs + src/tui/session_picker/loading.rs (the latter already does deferred loading, but is bound by the underlying file scan).

Cross-references

• Session tree branching: /tree, /fork, /clone in-place navigation #2 Tree branching — both this issue and Session tree branching: /tree, /fork, /clone in-place navigation #2 bump the session schema; align so they land as one consistent v3 format.
• Add general-purpose jcode doctor diagnostic command #8 jcode doctor --only sessions should detect stale index and missing checkpoints.

Reference

• pi_agent_rust: Session Store V2 Sidecar (Large Session Fast-Path), Session Indexing.