Parent: #240
Context
Port the append-only JSONL ledger, content sidecar, lock protocol, and SQLite archive to Rust. Highest-risk crate after the parsers because the lock protocol and SQLite path have correctness invariants the rest of the system depends on.
Scope
JSONL append + content sidecar
- Port
packages/ledger/src/file-adapter.ts and packages/ledger/src/index-sidecar.ts (the per-line JSON.parse loop at index-sidecar.ts:188-203).
- Port the hot loop at
packages/ledger/src/archive.ts:1237-1266 (createReadStream → JSON.parse per line) using serde_json::Deserializer::from_reader().into_iter().
Lock protocol — replicate behavior, not primitives
Do NOT use fs2::FileExt::lock_exclusive (flock(2)). It has different cross-process semantics on macOS/Windows and would silently change behavior. Replicate the TS protocol from packages/ledger/src/adapters/file-lock.ts:89-138 exactly:
- Exclusive-file-creation (
open(lp, 'wx') equivalent — Rust OpenOptions::new().write(true).create_new(true).open(lp)).
- Two-phase retry: 50×20ms (1s fast) then 40×250ms (10s slow).
- Orphan recovery: unlink lockfiles older than 5s when the held lock times out, then race the open again.
- Re-entrancy on lock name via task-local storage equivalent to TS
AsyncLocalStorage.
- Lock names in use:
archive, ledger-index, ledger, test-serialize.
SQLite archive
Port packages/ledger/src/archive.ts:700-914 using rusqlite. TS already wraps the entire tail in a single BEGIN/COMMIT — don't "fix" what isn't broken. Real perf wins:
- Multi-row
VALUES (...), (...) inserts to amortize statement overhead vs per-row prepared statements.
WAL mode + synchronous=NORMAL for the rebuild path.
- Prepared-statement reuse across tables instead of per-table
.run() loops.
Schema compatibility is non-negotiable. Existing archive.sqlite files in the wild must keep working without a forced rebuild — or ship a one-time migration in burn state rebuild archive --full.
Files
packages/ledger/src/file-adapter.ts
packages/ledger/src/index-sidecar.ts:188-203
packages/ledger/src/archive.ts:700-914 (sqlite write tx)
packages/ledger/src/archive.ts:1237-1266 (hot parse loop)
packages/ledger/src/adapters/file-lock.ts:89-138 (lock protocol — copy semantics)
packages/ledger/src/lock.ts (the withLock wrapper)
packages/ledger/src/schema.ts (LedgerLine, TurnLine, StampLine)
Depends on
#241, #242
Acceptance
Parent: #240
Context
Port the append-only JSONL ledger, content sidecar, lock protocol, and SQLite archive to Rust. Highest-risk crate after the parsers because the lock protocol and SQLite path have correctness invariants the rest of the system depends on.
Scope
JSONL append + content sidecar
packages/ledger/src/file-adapter.tsandpackages/ledger/src/index-sidecar.ts(the per-lineJSON.parseloop atindex-sidecar.ts:188-203).packages/ledger/src/archive.ts:1237-1266(createReadStream → JSON.parseper line) usingserde_json::Deserializer::from_reader().into_iter().Lock protocol — replicate behavior, not primitives
Do NOT use
fs2::FileExt::lock_exclusive(flock(2)). It has different cross-process semantics on macOS/Windows and would silently change behavior. Replicate the TS protocol frompackages/ledger/src/adapters/file-lock.ts:89-138exactly:open(lp, 'wx')equivalent — RustOpenOptions::new().write(true).create_new(true).open(lp)).AsyncLocalStorage.archive,ledger-index,ledger,test-serialize.SQLite archive
Port
packages/ledger/src/archive.ts:700-914usingrusqlite. TS already wraps the entire tail in a singleBEGIN/COMMIT— don't "fix" what isn't broken. Real perf wins:VALUES (...), (...)inserts to amortize statement overhead vs per-row prepared statements.WALmode +synchronous=NORMALfor the rebuild path..run()loops.Schema compatibility is non-negotiable. Existing
archive.sqlitefiles in the wild must keep working without a forced rebuild — or ship a one-time migration inburn state rebuild archive --full.Files
packages/ledger/src/file-adapter.tspackages/ledger/src/index-sidecar.ts:188-203packages/ledger/src/archive.ts:700-914(sqlite write tx)packages/ledger/src/archive.ts:1237-1266(hot parse loop)packages/ledger/src/adapters/file-lock.ts:89-138(lock protocol — copy semantics)packages/ledger/src/lock.ts(thewithLockwrapper)packages/ledger/src/schema.ts(LedgerLine,TurnLine,StampLine)Depends on
#241, #242
Acceptance
cargo test -p relayburn-ledgergreen.withLock("archive", …)callers all serialize correctly with no missed acquisitions; orphan recovery test artificially stales a lockfile and confirms recovery.archive.sqlitefrom a 1.5GB ledger opens in the Rust crate and answers queries without a rebuild.burn state rebuild archive --full(via Rust) on a 1.5GB ledger completes in ≤10s on M-series silicon.