Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e2b4620
feat: implement snapshot bootstrapper with cbor download + omnibus ch…
lowhung Nov 21, 2025
dcaab64
feat: enhance snapshot bootstrapper with error handling and configura…
lowhung Nov 21, 2025
e24aef2
Merge branch 'main' into lowhung/395-bootstrap-orchestration
lowhung Nov 21, 2025
25c2490
add todos back
lowhung Nov 21, 2025
8e73547
feat: add configuration files for snapshot management
lowhung Nov 21, 2025
e486a97
feat: refactor snapshot bootstrapper with improved config handling an…
lowhung Nov 21, 2025
deb40ca
feat: enhance snapshot download functionality with client reuse and i…
lowhung Nov 22, 2025
a44bb82
feat: update NOTES.md with configuration details and bootstrapping se…
lowhung Nov 22, 2025
be94615
feat: update dependencies and add ProgressReader for download tracking
lowhung Nov 23, 2025
a0175f8
feat: add headers and nonces JSON files, enhance HTTP client configur…
lowhung Nov 23, 2025
724d8ec
feat: split bootstrapper into a few modules with clearly defined resp…
lowhung Nov 23, 2025
abd5cd4
feat: simplify error handling in downloader by using std::io::Error::…
lowhung Nov 23, 2025
6d445e5
feat: simplify error handling in downloader by using std::io::Error::…
lowhung Nov 23, 2025
de9e6aa
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
03fa02f
feat: rename error variant for clarity and improve error handling in …
lowhung Nov 24, 2025
8b72d9d
feat: rename error variant for clarity and improve error handling in …
lowhung Nov 24, 2025
a740fae
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
26088e9
feat: add new error variant for stream read failures in downloader
lowhung Nov 24, 2025
5c51081
feat: simplify error handling for stream reading in downloader
lowhung Nov 24, 2025
da2e4ac
feat: add wiremock and flate2 dependencies for downloader tests
lowhung Nov 24, 2025
91bdfb1
feat: reorganize module structure and update dependencies for snapsho…
lowhung Nov 24, 2025
9efcf06
feat: refactor module structure and update dependencies for downloade…
lowhung Nov 24, 2025
708088c
feat: update workspace dependencies for caryatid_process and config i…
lowhung Nov 24, 2025
cec71c4
feat: update workspace dependencies for caryatid_process and config i…
lowhung Nov 24, 2025
601c31f
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
cd8e04c
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
43c2875
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
5d41589
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
061780b
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 24, 2025
084bf46
Refactor snapshot bootstrapper configuration and update topic naming
lowhung Nov 25, 2025
15e382f
Refactor snapshot bootstrapper configuration and update topic naming
lowhung Nov 25, 2025
383c0c9
Merge remote-tracking branch 'origin/lowhung/395-bootstrap-orchestrat…
lowhung Nov 25, 2025
35f4301
Update NOTES.md to clarify snapshot bootstrapping process and data re…
lowhung Nov 25, 2025
5b3bd79
Merge branch 'main' into lowhung/395-bootstrap-orchestration
lowhung Nov 25, 2025
a982790
chore: move tempfile to dev deps
lowhung Nov 25, 2025
c971d8c
feat: take stab at building block info to hand off
lowhung Nov 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 69 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 6 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,23 @@ to communicate between micro-services.

```mermaid
graph TB
subgraph Process A
Module1(Module 1)
Module2(Module 2)
Caryatid1(Caryatid Framework)
Module1 <--> Caryatid1
Module2 <--> Caryatid1
end
subgraph Process B
Module3(Module 3)
Caryatid2(Caryatid Framework)
Module3 <--> Caryatid2
end
RabbitMQ([RabbitMQ Message Bus])
style RabbitMQ fill:#eff
style RabbitMQ fill: #eff
Caryatid1 <--> RabbitMQ
Caryatid2 <--> RabbitMQ
```
Expand All @@ -61,6 +56,9 @@ graph TB
Fetches a chain snapshot from Mithril and replays all the blocks in it
- [Genesis Bootstrapper](modules/genesis_bootstrapper) - reads the Genesis
file for a chain and generates initial UTXOs
- [Snapshot Bootstrapper](modules/snapshot_bootstrapper) - downloads ledger state snapshot files for configured epochs,
streams and parses the CBOR data (UTXOs, pools, accounts, DReps, proposals), and publishes completion messages to
signal snapshot readiness to other modules.
- [Block Unpacker](modules/block_unpacker) - unpacks received blocks
into individual transactions
- [Tx Unpacker](modules/tx_unpacker) - parses transactions and generates UTXO
Expand All @@ -69,7 +67,8 @@ graph TB
- [SPO State](modules/spo_state) - matches SPO registrations and retirements
- [DRep State](modules/drep_state) - tracks DRep registrations
- [Governance State](modules/governance_state) - tracks Governance Actions and voting
- [Stake Delta Filter](modules/stake_delta_filter) - filters out stake address changes and handles stake pointer references
- [Stake Delta Filter](modules/stake_delta_filter) - filters out stake address changes and handles stake pointer
references
- [Epochs State](modules/epochs_state) - track fees blocks minted and epochs history
- [Accounts State](modules/accounts_state) - stake and reward accounts tracker
- [Assets State](modules/assets_state) - tracks native asset supply, metadata, transactions, and addresses
Expand Down
5 changes: 3 additions & 2 deletions common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,10 @@ thiserror = "2.0.17"
sha2 = "0.10.8"

[dev-dependencies]
caryatid_process = { workspace = true }
tempfile = "3.23"
config = { workspace = true }
tempfile = "3"
caryatid_process = { workspace = true }
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cargo shear chore, these are only being used in tests within common/src/resolver.rs



[lib]
crate-type = ["rlib"]
Expand Down
92 changes: 0 additions & 92 deletions common/src/snapshot/NOTES.md

This file was deleted.

15 changes: 6 additions & 9 deletions common/src/snapshot/streaming_snapshot.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1568,7 +1568,7 @@ impl StreamingSnapshotParser {
if utxo_count.is_multiple_of(1000000) {
let buffer_usage = buffer.len();
info!(
" Streamed {} UTXOs, buffer: {} MB, max entry: {} bytes",
"Streamed {} UTXOs, buffer: {} MB, max entry: {} bytes",
utxo_count,
buffer_usage / 1024 / 1024,
max_single_entry_size
Expand Down Expand Up @@ -1625,20 +1625,17 @@ impl StreamingSnapshotParser {
}
}

info!(" 🎯 STREAMING RESULTS:");
info!(" UTXOs processed: {}", utxo_count);
info!("Streaming results:");
info!(" UTXOs processed: {}", utxo_count);
info!(
" Total data streamed: {:.2} MB",
" Total data streamed: {:.2} MB",
total_bytes_processed as f64 / 1024.0 / 1024.0
);
info!(
" Peak buffer usage: {} MB (vs 2.1GB before!)",
" Peak buffer usage: {} MB",
PARSE_BUFFER_SIZE / 1024 / 1024
);
info!(
" • Largest single entry: {} bytes",
max_single_entry_size
);
info!(" Largest single entry: {} bytes", max_single_entry_size);

Ok(utxo_count)
}
Expand Down
6 changes: 4 additions & 2 deletions modules/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ compose the Acropolis Architecture
Fetches a chain snapshot from Mithril and replays all the blocks in it
* [Genesis Bootstrapper](genesis_bootstrapper) - reads the Genesis
file for a chain and generates initial UTXOs
* [Snapshot Bootstrapper](snapshot_bootstrapper) - downloads ledger state snapshot files for configured epochs,
streams and parses the CBOR data (UTXOs, pools, accounts, DReps, proposals), and publishes completion messages to
signal snapshot readiness to other modules.
* [Block Unpacker](block_unpacker) - unpacks received blocks
into individual transactions
* [Tx Unpacker](tx_unpacker) - parses transactions and generates UTXO
Expand All @@ -19,7 +22,6 @@ compose the Acropolis Architecture
* [DRep State](drep_state) - tracks DRep registrations
* [Governance State](governance_state) - tracks Governance Actions and voting
* [Stake Delta Filter](stake_delta_filter) - filters out stake address changes and handles stake pointer references
* [Epoch Activity Counter](epoch_activity_couinter) - counts fees and block production for rewards
* [Accounts State](accounts_state) - stake and reward accounts tracker

## How to add a new module
Expand Down Expand Up @@ -88,7 +90,7 @@ to call `MyModule::register()` in the process `main()`:
use acropolis_module_my_module::MyModule;

// in main()...
MyModule::register(&mut process);
MyModule::register( & mut process);
```

You also need to mention the module in (e.g.) `omnibus.toml` to get it created, even if all
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -379,7 +379,7 @@ impl MithrilSnapshotFetcher {
// Send completion message
if let Some(last_block_info) = last_block_info {
info!(
"Finished shapshot at block {}, epoch {}",
"Finished snapshot at block {}, epoch {}",
last_block_info.number, last_block_info.epoch
);
let message_enum =
Expand Down
16 changes: 15 additions & 1 deletion modules/snapshot_bootstrapper/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,20 @@ anyhow = { workspace = true }
config = { workspace = true }
tokio = { workspace = true }
tracing = { workspace = true }
serde = { workspace = true, features = ["rc"] }
serde_json = { workspace = true }
thiserror = "2.0.17"
async-compression = { version = "0.4.32", features = ["tokio", "gzip"] }
reqwest = { version = "0.12", features = ["stream"] }
futures-util = "0.3.31"
tokio-util = "0.7.17"
hex = "0.4.3"

[dev-dependencies]
wiremock = "0.6.5"
flate2 = "1.1.5"
tempfile = "3.23.0"


[lib]
path = "src/snapshot_bootstrapper.rs"
path = "src/bootstrapper.rs"
Loading