store: move SyncNotes loop into the database layer#2027
Open
bigeez wants to merge 3 commits into0xMiden:nextfrom
Open
store: move SyncNotes loop into the database layer#2027bigeez wants to merge 3 commits into0xMiden:nextfrom
bigeez wants to merge 3 commits into0xMiden:nextfrom
Conversation
Replace the N-round-trip loop in `State::sync_notes` with a single `get_note_sync_multi` call that returns all matching blocks in one database transaction. Previously each call to `get_note_sync` used a subquery to find the minimum matching block number, returned only that block's notes, and required the caller to loop and advance `current_from` on every iteration. For a response spanning N blocks this meant N async `transact` calls through tokio's blocking thread pool. The new `select_notes_since_block_by_tag` query drops the inner `WHERE block_num = (SELECT MIN(...))` constraint and returns all matching notes ordered by block number. `get_note_sync_multi` then groups them by block via a `BTreeMap` (preserving ascending order) and fetches each block header within the same transaction. The application layer in `sync_notes` now iterates already-fetched data to enforce the payload cap, instead of driving repeated DB round trips. Also removes the unused `account_ids` parameter from the note sync query path — it was always passed as an empty slice. Closes 0xMiden#1871
- Add missing `BTreeMap` import in notes.rs (fixes build) - Split import group in sync_state.rs onto separate lines (fixes nightly rustfmt) - Add changelog entry for 0xMiden#2027
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1871
Summary
Replaces the N async round-trip loop in
State::sync_noteswith a singleget_note_sync_multicall that performs note-sync pagination inside one database transaction.Before:
sync_notesdrove a loop callingget_note_syncon each iteration. Each call opened a new asynctransacton tokio's blocking thread pool, ran a two-stage SQL query to find the next matching block and fetch that block's notes, then returned a single block. For a response spanning N blocks this meant N blocking task dispatches.After:
State::sync_notesmakes one async DB call. Inside that transaction,get_note_sync_multirepeatedly fetches the next matching block, appends it while it fits within the response payload budget, and stops before returning blocks that would exceed the cap. This preserves response pagination/backpressure while avoiding repeated async DB transactions.Also removes the unused
account_idsparameter from the note sync query path. It was always passed as&[]but threaded through the entire call chain.Changes
queries::select_notes_since_block_by_tagreturns notes from the next matching block in the range, ordered by block/batch/note indexqueries::get_note_sync_multiloops within one transaction and returnsVec<NoteSyncUpdate>while respectingMAX_RESPONSE_PAYLOAD_BYTESDb::get_note_sync_multipasses the response payload cap into the transaction-level helperState::sync_noteskeeps the MMR-proof assembly and final payload guardTest plan
cargo test -p miden-node-store note_sync(blocked locally bymiden-core-lib:project 'miden-core' is missing its manifest path)note_sync_across_multiple_blocksverifies all 3 blocks are returned through one helper callnote_sync_multi_respects_payload_limitverifies returned blocks stop at the payload budgetnote_sync_no_matching_tagsasserts emptyVecinstead ofNone