feat: compute and persist blocks transactions block range roots

[Alenar]

Content

This PR:

• Define CardanoBlockTransactionMkTreeNode in mithril-common, a type that represent a leaf of the merkle tree for CardanoBlocksTransactions block ranges roots and can be sorted.
• Make ChainDataImporter in internal/mithril-cardano-node-chain compute and send to persistence the block ranges roots for CardanoBlocksTransactions
• Add necessary queries in internal/mithril-persistence to store/retrieve CardanoBlocksTransactions block ranges roots and retrieve CardanoBlockTransactionMkTreeNode

Important

The threshold for pruning data with the ChainDataImporterWithPruner had to be updated to take in account that there's now two tables for block ranges roots (one "legacy" for CardanoTransactions and one "new" for CardanoBlocksTransactions).

The pruner prune data that have already be computed into block ranges roots, but now with two table we have to ensure that this computation was done for both so the threshold was modified to be "the minimum between the highest start block number of the 'new' and the 'legacy' stored block range roots" (before it was simply "the highest start block number stored in the legacy block range roots").

Details

mithril-common

• Add CardanoBlockTransactionMkTreeNode:
  • Designed to be used a the MKTreeNode to compute block range roots for CardanoBlocksTransactions
  • With an manual implementation for ordering, to allow sort rust side after data retrieval in db
• Add MKTree::new_from_iter to allow construction from any IntoIterator, as the existing new works only with types that can be sliced, which is not the case of BTreeSet (the collection used when retrieving CardanoBlockTransactionMkTreeNode from the database).

mithril-cardano-node-chain

• ChainDataStore: add get_highest_block_range, get_blocks_and_transactions_in_range, and store_block_range_roots
• BlockRangeImporter:
  • rename run method to run_legacy
  • add new run that compute block ranges roots based on stored CardanoBlocksTransactions

mithril-persistence

• New CardanoBlockTransactionsRecord
• Add queries to fetch CardanoBlockTransactionsRecord from the sqlite database
• Add queries to fetch, insert, and delete BlockRangeRootRecord computed from CardanoBlocksTransactions and stored in the block_range_root table (copied from the existing 'legacy' queries)
• CardanoTransactionRepository:
  • Define the public methods that will be used to store and retrieve CardanoBlockTransactionsRecord and the BlockRangeRootRecord for CardanoBlocksTransactions
  • rename get_highest_start_block_number_for_legacy_block_range_roots to get_prune_blocks_threshold and update it in the light of the fact that there's two block range roots tables.

Pre-submit checklist

• Branch
  • Tests are provided (if possible)
  • Crates versions are updated (if relevant)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
• PR
  • All check jobs of the CI have succeeded
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested
• Documentation
  • No new TODOs introduced

Issue(s)

Relates to #2908

[github-actions]

Test Results

    5 files  ±  0    172 suites  ±0   39m 18s ⏱️ - 1m 35s
2 577 tests + 64  2 577 ✅ + 64  0 💤 ±0  0 ❌ ±0 
8 039 runs  +256  8 039 ✅ +256  0 💤 ±0  0 ❌ ±0 

Results for commit 212ec7b. ± Comparison against base commit e3edf71.

This pull request removes 10 and adds 74 tests. Note that renamed tests count towards both.

mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_range_root_retrieves_only_strictly_required_transactions
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::can_compute_block_ranges_even_if_last_blocks_in_range_dont_have_transactions
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::can_compute_block_ranges_up_to_the_strict_end_of_a_block_range
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::compute_block_range_merkle_root
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::if_all_block_ranges_computed_nothing_computed_and_stored
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::if_half_block_ranges_are_stored_the_other_half_is_computed_and_stored
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::if_nothing_stored_parse_and_store_all_block_ranges
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::if_theres_gap_between_two_stored_block_ranges_it_can_still_compute_their_root
mithril-cardano-node-chain ‑ test::double::chain_data_store::tests::get_transactions_in_range_returns_empty_when_no_transactions
mithril-persistence ‑ database::repository::cardano_transaction_repository::tests::get_highest_start_block_number_for_legacy_block_range_roots

mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::block_range_root_retrieves_only_strictly_required_blocks_and_transactions
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::can_compute_block_ranges_even_if_last_blocks_in_range_dont_have_transactions
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::can_compute_block_ranges_up_to_the_strict_end_of_a_block_range
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::compute_block_range_merkle_root
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::if_all_block_ranges_computed_nothing_computed_and_stored
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::if_half_block_ranges_are_stored_the_other_half_is_computed_and_stored
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::if_nothing_stored_parse_and_store_all_block_ranges
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::block_ranges::if_theres_gap_between_two_stored_block_ranges_it_can_still_compute_their_root
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::legacy_block_ranges::block_range_root_retrieves_only_strictly_required_transactions
mithril-cardano-node-chain ‑ chain_importer::block_ranges_importer::tests::legacy_block_ranges::can_compute_block_ranges_even_if_last_blocks_in_range_dont_have_transactions
…

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

This PR adds support for computing and persisting block-range Merkle roots for CardanoBlocksTransactions (in addition to the existing legacy CardanoTransactions roots), including new leaf types, persistence queries/records, and importer logic that computes + stores both root sets.

Changes:

• Introduce CardanoBlockTransactionMkTreeNode and related conversions to generate sortable Merkle leaves for blocks + transactions.
• Extend the chain importer/store API to compute & persist new block-range roots, while still computing legacy roots.
• Add persistence-layer records + queries for the new block_range_root table and for retrieving blocks-with-transactions projections.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
mithril-common/src/entities/cardano_block.rs	Adds CardanoBlockTransactionMkTreeNode, leaf identifiers, and ordering for deterministic Merkle leaves.
mithril-common/src/crypto_helper/merkle_tree.rs	Adds MKTree::new_from_iter to build trees from iterators (used by new block/tx leaf sets).
internal/cardano-node/mithril-cardano-node-chain/src/chain_importer/api.rs	Extends ChainDataStore with APIs to fetch block+tx leaves and store new block-range roots.
internal/cardano-node/mithril-cardano-node-chain/src/chain_importer/block_ranges_importer.rs	Computes/stores new CardanoBlocksTransactions block-range roots and still computes legacy roots.
internal/cardano-node/mithril-cardano-node-chain/src/chain_importer/service.rs	Runs both new and legacy block-range root computations during import.
internal/cardano-node/mithril-cardano-node-chain/src/test/double/chain_data_store.rs	Updates in-memory store + tests for new block-range roots and block+tx retrieval.
internal/mithril-persistence/src/database/repository/cardano_transaction_repository.rs	Adds new root CRUD and retrieval of blocks-with-transactions; updates prune threshold logic to consider both root tables.
internal/mithril-persistence/src/database/record/cardano_block_transactions.rs	New record type for hydrated blocks + aggregated tx hashes and conversion to Merkle leaves.
internal/mithril-persistence/src/database/record/mod.rs	Exports new CardanoBlockTransactionsRecord.
internal/mithril-persistence/src/database/query/mod.rs	Wires in new block-range-root query module.
internal/mithril-persistence/src/database/query/block_range_root/*	Adds insert/get/delete queries for the new block_range_root table.
internal/mithril-persistence/src/database/query/cardano_block/mod.rs	Exposes new “get blocks with txs” query.
internal/mithril-persistence/src/database/query/cardano_block/get_cardano_block_with_transactions.rs	Adds query to fetch blocks and group-concat their tx hashes.
mithril-aggregator/src/database/repository/cardano_transaction_repository.rs	Implements new ChainDataStore methods via persistence repository.
mithril-signer/src/database/repository/cardano_transaction_repository.rs	Implements new ChainDataStore methods via persistence repository.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[turmelclem]

LGTM 👍

[jpraynaud]

LGTM 🔥

Great work!