Skip to content

[Rust port] relayburn-ledger: JSONL + lock + sqlite archive #243

@willwashburn

Description

@willwashburn

Parent: #240

Context

Port the append-only JSONL ledger, content sidecar, lock protocol, and SQLite archive to Rust. Highest-risk crate after the parsers because the lock protocol and SQLite path have correctness invariants the rest of the system depends on.

Scope

JSONL append + content sidecar

  • Port packages/ledger/src/file-adapter.ts and packages/ledger/src/index-sidecar.ts (the per-line JSON.parse loop at index-sidecar.ts:188-203).
  • Port the hot loop at packages/ledger/src/archive.ts:1237-1266 (createReadStream → JSON.parse per line) using serde_json::Deserializer::from_reader().into_iter().

Lock protocol — replicate behavior, not primitives

Do NOT use fs2::FileExt::lock_exclusive (flock(2)). It has different cross-process semantics on macOS/Windows and would silently change behavior. Replicate the TS protocol from packages/ledger/src/adapters/file-lock.ts:89-138 exactly:

  • Exclusive-file-creation (open(lp, 'wx') equivalent — Rust OpenOptions::new().write(true).create_new(true).open(lp)).
  • Two-phase retry: 50×20ms (1s fast) then 40×250ms (10s slow).
  • Orphan recovery: unlink lockfiles older than 5s when the held lock times out, then race the open again.
  • Re-entrancy on lock name via task-local storage equivalent to TS AsyncLocalStorage.
  • Lock names in use: archive, ledger-index, ledger, test-serialize.

SQLite archive

Port packages/ledger/src/archive.ts:700-914 using rusqlite. TS already wraps the entire tail in a single BEGIN/COMMIT — don't "fix" what isn't broken. Real perf wins:

  • Multi-row VALUES (...), (...) inserts to amortize statement overhead vs per-row prepared statements.
  • WAL mode + synchronous=NORMAL for the rebuild path.
  • Prepared-statement reuse across tables instead of per-table .run() loops.

Schema compatibility is non-negotiable. Existing archive.sqlite files in the wild must keep working without a forced rebuild — or ship a one-time migration in burn state rebuild archive --full.

Files

  • packages/ledger/src/file-adapter.ts
  • packages/ledger/src/index-sidecar.ts:188-203
  • packages/ledger/src/archive.ts:700-914 (sqlite write tx)
  • packages/ledger/src/archive.ts:1237-1266 (hot parse loop)
  • packages/ledger/src/adapters/file-lock.ts:89-138 (lock protocol — copy semantics)
  • packages/ledger/src/lock.ts (the withLock wrapper)
  • packages/ledger/src/schema.ts (LedgerLine, TurnLine, StampLine)

Depends on

#241, #242

Acceptance

  • cargo test -p relayburn-ledger green.
  • Lock-protocol property test: 100 concurrent withLock("archive", …) callers all serialize correctly with no missed acquisitions; orphan recovery test artificially stales a lockfile and confirms recovery.
  • Cross-process lock test: a Rust-held lock blocks a TS-held lock and vice versa (run TS adapter and Rust adapter against the same lockfile).
  • Existing archive.sqlite from a 1.5GB ledger opens in the Rust crate and answers queries without a rebuild.
  • burn state rebuild archive --full (via Rust) on a 1.5GB ledger completes in ≤10s on M-series silicon.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions