Context
crates/relayburn-sdk/src/ledger/reader.rs:32-53 (query_turns) and :121-142 (select_records) load every row of a table into Vec<String> and then filter in Rust. The Query filters are pure SQL predicates with existing indexes (idx_turns_ts, idx_turns_session):
since / until → WHERE rowid >= ? AND rowid <= ? (or whatever timestamp column maps to idx_turns_ts).session_id → WHERE session_id = ?.source → WHERE source = ?.
Every call we re-parse JSON for thousands of turns we then throw away. This is the biggest ledger perf win available.
Adjacent: no prepare_cached anywhere in the ledger. The TS sibling got prepared-statement caching free from better-sqlite3; the rusqlite port lost it. With burn ingest --watch firing ingest_all every second, every read goes through full prepare/teardown.
Proposed fix
- Build
WHERE/parameter pairs from Query and bind them via params!. Keep relationship_passes (currently does serde_json::to_value enum→string round-trips per row) but make the source comparison a single &'static str from a wire_str() method rather than per-row Value alloc — see #(forthcoming enum-string-conversion issue).
- Switch every hot SELECT in
ledger/reader.rs to Connection::prepare_cached. Writers can stay one-shot inside their own transaction since they're already amortized.
- Consider gating
collect_stamps (ledger/reader.rs:144-176) on a Query flag — today it folds every stamp into every turn, even when the caller never reads enrichment.
Verification
- Conformance gate (deep-equal vs TS
@relayburn/sdk) must keep passing.
- Add a benchmark over a large fixture — expected speedup is order-of-magnitude on
--since queries against a many-month ledger.
References
crates/relayburn-sdk/src/ledger/reader.rs:32-53, 121-142, 144-176, 269-277, 365- Companion fingerprint hot-path:
crates/relayburn-sdk/src/ledger/fingerprint.rs:118 allocates a Value per record just to read out the source string.
Context
crates/relayburn-sdk/src/ledger/reader.rs:32-53(query_turns) and:121-142(select_records) load every row of a table intoVec<String>and then filter in Rust. TheQueryfilters are pure SQL predicates with existing indexes (idx_turns_ts,idx_turns_session):since/until→WHERE rowid >= ? AND rowid <= ?(or whatever timestamp column maps toidx_turns_ts).session_id→WHERE session_id = ?.source→WHERE source = ?.Every call we re-parse JSON for thousands of turns we then throw away. This is the biggest ledger perf win available.
Adjacent: no
prepare_cachedanywhere in the ledger. The TS sibling got prepared-statement caching free frombetter-sqlite3; the rusqlite port lost it. Withburn ingest --watchfiringingest_allevery second, every read goes through full prepare/teardown.Proposed fix
WHERE/parameter pairs fromQueryand bind them viaparams!. Keeprelationship_passes(currently doesserde_json::to_valueenum→string round-trips per row) but make the source comparison a single&'static strfrom awire_str()method rather than per-row Value alloc — see #(forthcoming enum-string-conversion issue).ledger/reader.rstoConnection::prepare_cached. Writers can stay one-shot inside their own transaction since they're already amortized.collect_stamps(ledger/reader.rs:144-176) on aQueryflag — today it folds every stamp into every turn, even when the caller never reads enrichment.Verification
@relayburn/sdk) must keep passing.--sincequeries against a many-month ledger.References
crates/relayburn-sdk/src/ledger/reader.rs:32-53, 121-142, 144-176, 269-277, 365crates/relayburn-sdk/src/ledger/fingerprint.rs:118allocates a Value per record just to read out the source string.