-
Notifications
You must be signed in to change notification settings - Fork 1
feat(benches): PR-12 — snapshot hash microbench (Closes #43) #113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
a8ffc92
docs: regen rollup after benches update
flyingrobots f8ee3c9
bench(deps): add blake3 dev-dep for deterministic rule id; fmt
flyingrobots 72d588c
Merge remote-tracking branch 'origin/main' into echo/pr-12-snapshot-b…
flyingrobots c66c519
merge: resolve benches Cargo.toml conflict (keep version-pinned rmg-c…
flyingrobots b4b58e4
docs(guard): refresh echo-total rollup
flyingrobots 748f960
bench(deps): pin blake3 = 1.8.2 in rmg-benches (cargo-deny policy)
flyingrobots ab56f24
bench(snapshot_hash): precompute link type id and fix edge labels e-i…
flyingrobots 1e85ead
bench(scheduler_drain): return Vec<NodeId>; avoid re-hashing make_nod…
flyingrobots f54d4a2
docs: refresh echo-total rollup for PR-113
flyingrobots 7e9dbdf
docs(guard): record PR-12 benches pin and micro-optimizations; refres…
flyingrobots b6b58b5
bench: restore allow(missing_docs) for criterion macro helpers in ben…
flyingrobots c0ec43f
bench: snapshot_hash/scheduler_drain polish per review; docs guard up…
flyingrobots c44c827
docs(rollup): refresh echo-total after benches README + links
flyingrobots File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| # Echo Benches (rmg-benches) | ||
|
|
||
| This crate hosts Criterion microbenchmarks for Echo’s Rust core (`rmg-core`). | ||
|
|
||
| Benchmarks are executable documentation of performance. Each bench includes | ||
| module-level docs describing what is measured, why, and how to interpret | ||
| results. This README summarizes how to run them and read the output. | ||
|
|
||
| ## What’s Here | ||
|
|
||
| - `snapshot_hash.rs` | ||
| - Builds a linear chain of `n` entities reachable from `root` and measures | ||
| the snapshot (state_root) hash of the reachable subgraph. | ||
| - Throughput “elements” = nodes in the reachable set (`n` entities + 1 root). | ||
| - Sizes: `10`, `100`, `1000` to show order-of-magnitude scaling without long | ||
| runtimes. | ||
|
|
||
| - `scheduler_drain.rs` | ||
| - Registers a trivial no-op rule and applies it to `n` entity nodes within a | ||
| transaction to focus on scheduler overhead (not executor work). | ||
| - Throughput “elements” = rule applications (`n`). Uses `BatchSize::PerIteration` | ||
| so engine construction is excluded from timing. | ||
|
|
||
| ## Run | ||
|
|
||
| Run the full benches suite: | ||
|
|
||
| ``` | ||
| cargo bench -p rmg-benches | ||
| ``` | ||
|
|
||
| Run a single bench target (faster dev loop): | ||
|
|
||
| ``` | ||
| cargo bench -p rmg-benches --bench snapshot_hash | ||
| cargo bench -p rmg-benches --bench scheduler_drain | ||
| ``` | ||
|
|
||
| Criterion HTML reports are written under `target/criterion/<group>/report/index.html`. | ||
|
|
||
| ## Interpreting Results | ||
|
|
||
| - Use the throughput value to sanity‑check the scale of work per iteration. | ||
| - The primary signal is `time/iter` across inputs (e.g., 10 vs 100 vs 1000). | ||
| - For regressions, compare runs in `target/criterion` or host an artifact in CI | ||
| (planned for PR‑14/15) and gate on percent deltas. | ||
|
|
||
| ## Environment Notes | ||
|
|
||
| - Toolchain: `stable` Rust (see `rust-toolchain.toml`). | ||
| - Dependency policy: avoid wildcards; benches use a minor pin for `blake3`. | ||
| - Repro: keep your machine under minimal background load; prefer `--quiet` and | ||
| close other apps. | ||
|
|
||
| ## Flamegraphs (optional) | ||
|
|
||
| If you have [`inferno`](https://github.com/jonhoo/inferno) or `cargo-flamegraph` | ||
| installed, you can profile a bench locally. Example (may require sudo on Linux): | ||
|
|
||
| ``` | ||
| cargo flamegraph -p rmg-benches --bench snapshot_hash -- --sample-size 50 | ||
| ``` | ||
|
|
||
| These tools are not required for CI and are optional for local analysis. | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| #![allow(missing_docs)] | ||
| //! Benchmark: scheduler drain throughput with a no-op rule | ||
| //! | ||
| //! Applies a trivial no-op rule across `n` entity nodes to measure scheduler | ||
| //! overhead rather than executor work. Construction happens in the setup phase; | ||
| //! measurement covers applying the rule to each node and committing a tx. | ||
| //! | ||
| //! Throughput "elements" are rule applications (`n`). | ||
| //! BatchSize::PerIteration ensures engine construction is excluded from timing. | ||
| //! | ||
| //! TODO(PR-14/15): Persist JSON artifacts and add a regression gate. | ||
| use blake3::Hasher; | ||
| use criterion::{criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput}; | ||
| use rmg_core::{ | ||
| make_node_id, make_type_id, ApplyResult, ConflictPolicy, Engine, Footprint, Hash, NodeId, | ||
| NodeRecord, PatternGraph, RewriteRule, | ||
| }; | ||
|
|
||
| // Bench constants to avoid magic strings. | ||
| const BENCH_NOOP_RULE_NAME: &str = "bench/noop"; | ||
| const RULE_ID_PREFIX: &[u8] = b"rule:"; | ||
| const ENTITY_TYPE_STR: &str = "entity"; | ||
| const ENT_LABEL_PREFIX: &str = "sched-ent-"; | ||
|
|
||
| fn bench_noop_rule() -> RewriteRule { | ||
| // Deterministic rule id: blake3("rule:" ++ name) | ||
| let id: Hash = { | ||
| let mut h = Hasher::new(); | ||
| h.update(RULE_ID_PREFIX); | ||
| h.update(BENCH_NOOP_RULE_NAME.as_bytes()); | ||
| h.finalize().into() | ||
| }; | ||
| fn matcher(_s: &rmg_core::GraphStore, _n: &rmg_core::NodeId) -> bool { | ||
| true | ||
| } | ||
| fn executor(_s: &mut rmg_core::GraphStore, _n: &rmg_core::NodeId) {} | ||
| fn footprint(_s: &rmg_core::GraphStore, _n: &rmg_core::NodeId) -> Footprint { | ||
| Footprint::default() | ||
| } | ||
| RewriteRule { | ||
| id, | ||
| name: BENCH_NOOP_RULE_NAME, | ||
| left: PatternGraph { nodes: vec![] }, | ||
| matcher, | ||
| executor, | ||
| compute_footprint: footprint, | ||
| factor_mask: 0, | ||
| conflict_policy: ConflictPolicy::Abort, | ||
| join_fn: None, | ||
| } | ||
| } | ||
|
|
||
| fn build_engine_with_entities(n: usize) -> (Engine, Vec<NodeId>) { | ||
| let mut engine = rmg_core::build_motion_demo_engine(); | ||
| // Register a no-op rule to isolate scheduler overhead from executor work. | ||
| engine | ||
| .register_rule(bench_noop_rule()) | ||
| .expect("Failed to register benchmark noop rule"); | ||
|
|
||
| let ty = make_type_id(ENTITY_TYPE_STR); | ||
| let mut ids = Vec::with_capacity(n); | ||
| for i in 0..n { | ||
| let label = format!("{}{}", ENT_LABEL_PREFIX, i); | ||
| let id = make_node_id(&label); | ||
| engine.insert_node(id, NodeRecord { ty, payload: None }); | ||
| ids.push(id); | ||
| } | ||
| (engine, ids) | ||
| } | ||
|
|
||
| fn bench_scheduler_drain(c: &mut Criterion) { | ||
| let mut group = c.benchmark_group("scheduler_drain"); | ||
| for &n in &[10usize, 100, 1_000] { | ||
| // Throughput: number of rule applications in this run (n entities). | ||
| group.throughput(Throughput::Elements(n as u64)); | ||
| group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| { | ||
| b.iter_batched( | ||
flyingrobots marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| || build_engine_with_entities(n), | ||
| |(mut engine, ids)| { | ||
| // Apply the no-op rule to all entities, then commit. | ||
| let tx = engine.begin(); | ||
| for id in &ids { | ||
| let res = engine | ||
| .apply(tx, BENCH_NOOP_RULE_NAME, id) | ||
| .expect("Failed to apply noop bench rule"); | ||
| // Avoid affecting timing; check only in debug builds. | ||
| debug_assert!(matches!(res, ApplyResult::Applied)); | ||
| } | ||
| let snap = engine.commit(tx).expect("Failed to commit benchmark tx"); | ||
| // Ensure the commit work is not optimized away. | ||
| criterion::black_box(snap); | ||
flyingrobots marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }, | ||
| BatchSize::PerIteration, | ||
| ) | ||
| }); | ||
| } | ||
| group.finish(); | ||
| } | ||
|
|
||
| criterion_group!(benches, bench_scheduler_drain); | ||
| criterion_main!(benches); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| #![allow(missing_docs)] | ||
| //! Benchmark: snapshot hash over a linear chain graph | ||
| //! | ||
| //! Builds a chain of `n` entities reachable from a single root node and | ||
| //! measures the cost of computing the snapshot (state_root) hash over the | ||
| //! reachable subgraph. Sizes (10, 100, 1000) provide an order-of-magnitude | ||
| //! progression to observe scaling trends without long runtimes. | ||
| //! | ||
| //! Throughput "elements" are the number of nodes in the reachable set | ||
| //! (n entities + 1 root). | ||
| //! | ||
| //! TODO(PR-14/15): Persist JSON artifacts and add a regression gate. | ||
| use criterion::{criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput}; | ||
| use rmg_core::{ | ||
| make_edge_id, make_node_id, make_type_id, EdgeRecord, Engine, GraphStore, NodeRecord, | ||
| }; | ||
|
|
||
| // String constants to avoid magic literals drifting silently. | ||
| const ROOT_ID_STR: &str = "root"; | ||
| const WORLD_TYPE_STR: &str = "world"; | ||
| const ENTITY_TYPE_STR: &str = "entity"; | ||
| const LINK_TYPE_STR: &str = "link"; | ||
| const ENT_LABEL_PREFIX: &str = "ent-"; | ||
|
|
||
| fn build_chain_engine(n: usize) -> Engine { | ||
| let mut store = GraphStore::default(); | ||
| let root = make_node_id(ROOT_ID_STR); | ||
| let world = make_type_id(WORLD_TYPE_STR); | ||
| store.insert_node( | ||
| root, | ||
| NodeRecord { | ||
| ty: world, | ||
| payload: None, | ||
| }, | ||
| ); | ||
| // Insert N nodes and connect them in a chain so all are reachable. | ||
| let entity_ty = make_type_id(ENTITY_TYPE_STR); | ||
| let link_ty = make_type_id(LINK_TYPE_STR); | ||
| let mut chain_tail = root; | ||
| for i in 0..n { | ||
| let to_label = format!("{}{}", ENT_LABEL_PREFIX, i); | ||
| let id = make_node_id(&to_label); | ||
| store.insert_node( | ||
| id, | ||
| NodeRecord { | ||
| ty: entity_ty, | ||
| payload: None, | ||
| }, | ||
| ); | ||
| // Human-friendly edge id: <from>-to-<to>. | ||
| let from_label = if i == 0 { | ||
| ROOT_ID_STR.to_string() | ||
| } else { | ||
| format!("{}{}", ENT_LABEL_PREFIX, i - 1) | ||
| }; | ||
| let edge_id = make_edge_id(&format!("edge-{}-to-{}", from_label, to_label)); | ||
| store.insert_edge( | ||
| chain_tail, | ||
| EdgeRecord { | ||
| id: edge_id, | ||
| from: chain_tail, | ||
| to: id, | ||
| ty: link_ty, | ||
| payload: None, | ||
| }, | ||
| ); | ||
| chain_tail = id; | ||
| } | ||
| Engine::new(store, root) | ||
| } | ||
|
|
||
| fn bench_snapshot_hash(c: &mut Criterion) { | ||
| let mut group = c.benchmark_group("snapshot_hash"); | ||
| for &n in &[10usize, 100, 1_000] { | ||
| // Throughput: total nodes in reachable set (n entities + 1 root). | ||
| group.throughput(Throughput::Elements(n as u64 + 1)); | ||
| group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| { | ||
| // Build engine in setup (not timed) and measure only hashing. | ||
| b.iter_batched( | ||
| || build_chain_engine(n), | ||
| |engine| { | ||
| let snap = engine.snapshot(); | ||
| criterion::black_box(snap.hash); | ||
| }, | ||
| BatchSize::SmallInput, | ||
| ) | ||
| }); | ||
flyingrobots marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| group.finish(); | ||
| } | ||
|
|
||
| criterion_group!(benches, bench_snapshot_hash); | ||
| criterion_main!(benches); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.