Skip to content

miner(loop): runLoop reads and JSON-parses the entire event ledger to obtain one seq number #10008

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

runLoop primes its ledger cursor by reading the whole ledger and taking the last element's seq
packages/loopover-miner/lib/loop-cli.ts:342:

  let sinceSeq = eventLedger.readEvents({}).at(-1)?.seq ?? 0;

readEvents with no filter runs SELECT * FROM miner_event_ledger ORDER BY seq ASC
(packages/loopover-miner/lib/event-ledger.ts:186, :231-233) and maps every row through rowToEntry
(packages/loopover-miner/lib/event-ledger.ts:126-135), which does a JSON.parse(row.payload_json) per row:

function rowToEntry(row: EventDbRow): LedgerEntry {
  return {
    id: row.id,
    seq: row.seq,
    type: row.event_type,
    repoFullName: row.repo_full_name,
    payload: JSON.parse(row.payload_json),
    createdAt: row.created_at,
  };
}

So every loopover-miner loop invocation materializes and JSON-parses the miner's entire append-only audit trail
in order to read a single integer, then discards all of it. The ledger is unbounded by default: retention is
opt-in and off unless an operator sets LOOPOVER_MINER_LEDGER_RETENTION_DAYS or
LOOPOVER_MINER_LEDGER_RETENTION_MAX_ROWS
(packages/loopover-miner/lib/store-maintenance.ts:8, :19-20, :146-156), and the module header describes the
ledger as "an immutable audit trail of every significant miner-loop event"
(packages/loopover-miner/lib/event-ledger.ts:14-17). A long-lived AMS box accumulates one row per discovered
issue, per plan step, per manage-poll snapshot, per PR outcome — and runLoop, the daemon meant to run
continuously, pays for all of them at startup.

The exact statement needed already exists inside the store, one line away from the fix —
packages/loopover-miner/lib/event-ledger.ts:180:

  const nextSeqStatement = db.prepare("SELECT COALESCE(MAX(seq), 0) + 1 AS nextSeq FROM miner_event_ledger");

seq is INTEGER NOT NULL UNIQUE (packages/loopover-miner/lib/event-ledger.ts:168), so SQLite has an index on
it and MAX(seq) is an index lookup, not a scan.

Every other consumer of the cursor already works on bounded, filtered reads:
buildLoopClosureSummary is called with { sinceSeq, repoFullName }
(packages/loopover-miner/lib/loop-cli.ts:600) and its result reassigns the cursor at :602. Only the priming
read at :342 is unbounded.

Requirements

  • EventLedger gains a latestSeq(): number method that returns the current MAX(seq), or 0 for an empty
    ledger, without materializing any row. Implement it with a SELECT COALESCE(MAX(seq), 0) AS latestSeq FROM miner_event_ledger statement prepared once at open time, alongside the existing statements at
    packages/loopover-miner/lib/event-ledger.ts:180-195.
  • packages/loopover-miner/lib/loop-cli.ts:342 uses eventLedger.latestSeq() and no longer calls
    readEvents({}).
  • latestSeq() must be exposed on the EventLedger type
    (packages/loopover-miner/lib/event-ledger.ts:45-51) and mirrored by a module-level convenience export next to
    appendEvent / readEvents (packages/loopover-miner/lib/event-ledger.ts:256-262), matching this module's
    existing default-ledger convention.
  • RunLoopOptions.initEventLedger (packages/loopover-miner/lib/loop-cli.ts:94) is an injection seam — every
    test double supplying an EventLedger must be updated so the loop still works with an injected ledger, and the
    new method must be part of the seam, not read off the concrete store type.
  • latestSeq() must return 0 for a ledger with no rows, so runLoop's existing ?? 0 semantics are preserved
    exactly and the first cycle's sinceSeq is unchanged.
  • After an appendEvent, latestSeq() must return the newly-appended entry's seq — the two must never
    disagree.
  • Do NOT change readEvents, appendEvent, rowToEntry, the retention path, or the ledger's immutability
    invariant (packages/loopover-miner/lib/event-ledger.ts:14-23).
  • Do NOT change the other readEvents() callers
    (packages/loopover-miner/lib/manage-status.ts:141, calibration-cli.ts, metrics-cli.ts,
    ams-calibration.ts, signal-tracking-store.ts) — they genuinely need the rows.

⚠️ Required pattern: add the prepared statement next to nextSeqStatement
(packages/loopover-miner/lib/event-ledger.ts:180) and return it from the same object literal the other methods
are defined on (packages/loopover-miner/lib/event-ledger.ts:197-248). What does NOT satisfy this issue:
(a) leaving readEvents({}) in place and slicing it (.slice(-1)), which still loads and parses every row;
(b) adding a limit/order option to readEvents and threading it through every caller, a change to the
ledger's whole read surface; (c) caching the seq in a module-level variable, which goes stale the moment a
sibling process appends.

Deliverables

  • EventLedger.latestSeq() exists in packages/loopover-miner/lib/event-ledger.ts, is declared on the
    EventLedger type, and is backed by a COALESCE(MAX(seq), 0) statement prepared at open time.
  • initEventLedger(":memory:").latestSeq() returns 0; after three appendEvent calls it returns 3 and
    equals the third entry's seq — asserted in test/unit/miner-event-ledger.test.ts.
  • runLoop no longer calls readEvents to prime sinceSeq: with an injected event ledger whose
    readEvents is a spy, a runLoop invocation that halts on the initial kill switch calls latestSeq() once
    and readEvents zero times — asserted in test/unit/miner-loop-cli.test.ts.
  • runLoop primes sinceSeq from latestSeq(): with an injected ledger reporting latestSeq() === 7, the
    first buildLoopClosureSummary call receives sinceSeq: 7 — asserted in
    test/unit/miner-loop-cli.test.ts.
  • With an empty ledger (latestSeq() === 0), the first buildLoopClosureSummary call receives
    sinceSeq: 0 — asserted in test/unit/miner-loop-cli.test.ts.
  • A regression test named for this bug (e.g. REGRESSION: runLoop primes its ledger cursor without reading every event) that fails against the current code.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example one that
adds latestSeq() to the ledger but leaves packages/loopover-miner/lib/loop-cli.ts:342 calling
readEvents({}) — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's coverage.include lists
packages/loopover-miner/lib/**/*.ts, so event-ledger.ts and loop-cli.ts are measured and gated. Every branch
the change introduces needs both arms tested: latestSeq() on an empty ledger (the COALESCE zero path) and on a
populated one; the module-level convenience export's lazily-opened default-ledger path
(packages/loopover-miner/lib/event-ledger.ts:251-254); and, in runLoop, both the initial-halt path (which
must still prime the cursor) and the normal path that reaches buildLoopClosureSummary.

Expected Outcome

Starting loopover-miner loop costs one indexed MAX(seq) lookup instead of loading and JSON-parsing the miner's
entire event ledger, so loop start-up time and memory stop growing with the length of the audit trail on
long-lived AMS boxes.

Links & Resources

  • packages/loopover-miner/lib/loop-cli.ts:342 — the unbounded priming read
  • packages/loopover-miner/lib/loop-cli.ts:600-602 — the only other cursor consumer, already bounded
  • packages/loopover-miner/lib/event-ledger.ts:126-135rowToEntry's per-row JSON.parse
  • packages/loopover-miner/lib/event-ledger.ts:180 — the existing MAX(seq) statement to mirror
  • packages/loopover-miner/lib/event-ledger.ts:186, :231-233 — the unfiltered SELECT *
  • packages/loopover-miner/lib/store-maintenance.ts:8, :146-156 — retention is opt-in, so the ledger is
    unbounded by default

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions