Skip to content

Update Readme explaining storage #149

Description

@willwashburn

Layer 1 — JSONL ledger (source of truth)

  • File: ~/.relayburn/ledger.jsonl (override with RELAYBURN_HOME)
  • Owner: @relayburn/ledger (packages/ledger/src/writer.ts)
  • Append-only, one LedgerLine per row. Schema is a tagged union
    (packages/ledger/src/schema.ts:74): turn, stamp, compaction,
    tool_result_event, user_turn, relationship.
  • Why JSONL: crash-safe append, no migrations, audit-friendly, every
    line is self-contained JSON. Pricing isn't stored — only usage tokens
    — so cost can be re-derived later.
  • Dedup: a sidecar index (.relayburn-index.jsonl) keys on message/turn
    IDs so re-ingestion is idempotent
    (packages/ledger/src/index-sidecar.ts).
  • Sister file: ~/.relayburn/content/.jsonl for full
    prompt/response bodies, separated so the ledger stays light.

Layer 2 — SQLite archive (derived read model)

  • File: ~/.relayburn/archive.sqlite
  • Built by buildArchive() in packages/ledger/src/archive.ts:399. WAL
    mode enabled.
  • Tables: sessions, turns, tool_calls, tool_result_events,
    compactions, archive_state. Indexed on the columns that actually get
    queried (ts, session_id, model, activity, workflow_id, fidelity).
  • Disposable. burn archive rebuild blows it away and reconstructs from
    the ledger.
  • Incremental: tracks a byte offset (ledger_offset_bytes) into the
    JSONL and only ingests the tail since last build.
  • Queries go through queryTurnsFromArchive()
    (packages/ledger/src/archive-query.ts:27).

How they relate

JSONL is canonical and durable; SQLite is a fast index for analytics
queries. They're kept in sync by an incremental cursor — the archive
is essentially a materialized view.

Recent commits (#82, #97, #128/#129) show a deliberate migration of
read paths (burn summary, MCP tools) from "scan the JSONL" to "query
the archive, building it lazily on each call". That's why the MCP
tools now call buildArchive() per invocation
(packages/mcp/src/tools/session-cost.ts:46) — it's cheap when
nothing's changed and guarantees fresh data when hooks have appended
new turns mid-session.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions