Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ It’s the core of the Echo engine: runtime, assets, networking, and tools all o

- Command: `cargo bench -p rmg-benches`
- Purpose: Runs Criterion micro-benchmarks for the benches crate (`crates/rmg-benches`).
- Location: see `crates/rmg-benches/` for sources and configuration.
- Docs: see `crates/rmg-benches/benches/README.md` for details, tips, and report paths.

### Core Principles

Expand Down
11 changes: 11 additions & 0 deletions crates/rmg-benches/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,23 @@ version = "0.1.0"
edition = "2021"
publish = false
license = "Apache-2.0"
description = "Microbenchmarks for Echo (rmg-core): snapshot hashing and scheduler throughput"

[dev-dependencies]
criterion = { version = "0.5", default-features = false, features = ["html_reports"] }
# Pin version alongside path to satisfy cargo-deny wildcard bans
rmg-core = { version = "0.1.0", path = "../rmg-core" }
# Minor-pin for semver compatibility; benches do not rely on a specific patch.
blake3 = "1.8"

[[bench]]
name = "motion_throughput"
harness = false

[[bench]]
name = "snapshot_hash"
harness = false

[[bench]]
name = "scheduler_drain"
harness = false
65 changes: 65 additions & 0 deletions crates/rmg-benches/benches/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Echo Benches (rmg-benches)

This crate hosts Criterion microbenchmarks for Echo’s Rust core (`rmg-core`).

Benchmarks are executable documentation of performance. Each bench includes
module-level docs describing what is measured, why, and how to interpret
results. This README summarizes how to run them and read the output.

## What’s Here

- `snapshot_hash.rs`
- Builds a linear chain of `n` entities reachable from `root` and measures
the snapshot (state_root) hash of the reachable subgraph.
- Throughput “elements” = nodes in the reachable set (`n` entities + 1 root).
- Sizes: `10`, `100`, `1000` to show order-of-magnitude scaling without long
runtimes.

- `scheduler_drain.rs`
- Registers a trivial no-op rule and applies it to `n` entity nodes within a
transaction to focus on scheduler overhead (not executor work).
- Throughput “elements” = rule applications (`n`). Uses `BatchSize::PerIteration`
so engine construction is excluded from timing.

## Run

Run the full benches suite:

```
cargo bench -p rmg-benches
```

Run a single bench target (faster dev loop):

```
cargo bench -p rmg-benches --bench snapshot_hash
cargo bench -p rmg-benches --bench scheduler_drain
```

Criterion HTML reports are written under `target/criterion/<group>/report/index.html`.

## Interpreting Results

- Use the throughput value to sanity‑check the scale of work per iteration.
- The primary signal is `time/iter` across inputs (e.g., 10 vs 100 vs 1000).
- For regressions, compare runs in `target/criterion` or host an artifact in CI
(planned for PR‑14/15) and gate on percent deltas.

## Environment Notes

- Toolchain: `stable` Rust (see `rust-toolchain.toml`).
- Dependency policy: avoid wildcards; benches use a minor pin for `blake3`.
- Repro: keep your machine under minimal background load; prefer `--quiet` and
close other apps.

## Flamegraphs (optional)

If you have [`inferno`](https://github.com/jonhoo/inferno) or `cargo-flamegraph`
installed, you can profile a bench locally. Example (may require sudo on Linux):

```
cargo flamegraph -p rmg-benches --bench snapshot_hash -- --sample-size 50
```

These tools are not required for CI and are optional for local analysis.

101 changes: 101 additions & 0 deletions crates/rmg-benches/benches/scheduler_drain.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#![allow(missing_docs)]
//! Benchmark: scheduler drain throughput with a no-op rule
//!
//! Applies a trivial no-op rule across `n` entity nodes to measure scheduler
//! overhead rather than executor work. Construction happens in the setup phase;
//! measurement covers applying the rule to each node and committing a tx.
//!
//! Throughput "elements" are rule applications (`n`).
//! BatchSize::PerIteration ensures engine construction is excluded from timing.
//!
//! TODO(PR-14/15): Persist JSON artifacts and add a regression gate.
use blake3::Hasher;
use criterion::{criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput};
use rmg_core::{
make_node_id, make_type_id, ApplyResult, ConflictPolicy, Engine, Footprint, Hash, NodeId,
NodeRecord, PatternGraph, RewriteRule,
};

// Bench constants to avoid magic strings.
const BENCH_NOOP_RULE_NAME: &str = "bench/noop";
const RULE_ID_PREFIX: &[u8] = b"rule:";
const ENTITY_TYPE_STR: &str = "entity";
const ENT_LABEL_PREFIX: &str = "sched-ent-";

fn bench_noop_rule() -> RewriteRule {
// Deterministic rule id: blake3("rule:" ++ name)
let id: Hash = {
let mut h = Hasher::new();
h.update(RULE_ID_PREFIX);
h.update(BENCH_NOOP_RULE_NAME.as_bytes());
h.finalize().into()
};
fn matcher(_s: &rmg_core::GraphStore, _n: &rmg_core::NodeId) -> bool {
true
}
fn executor(_s: &mut rmg_core::GraphStore, _n: &rmg_core::NodeId) {}
fn footprint(_s: &rmg_core::GraphStore, _n: &rmg_core::NodeId) -> Footprint {
Footprint::default()
}
RewriteRule {
id,
name: BENCH_NOOP_RULE_NAME,
left: PatternGraph { nodes: vec![] },
matcher,
executor,
compute_footprint: footprint,
factor_mask: 0,
conflict_policy: ConflictPolicy::Abort,
join_fn: None,
}
}

fn build_engine_with_entities(n: usize) -> (Engine, Vec<NodeId>) {
let mut engine = rmg_core::build_motion_demo_engine();
// Register a no-op rule to isolate scheduler overhead from executor work.
engine
.register_rule(bench_noop_rule())
.expect("Failed to register benchmark noop rule");

let ty = make_type_id(ENTITY_TYPE_STR);
let mut ids = Vec::with_capacity(n);
for i in 0..n {
let label = format!("{}{}", ENT_LABEL_PREFIX, i);
let id = make_node_id(&label);
engine.insert_node(id, NodeRecord { ty, payload: None });
ids.push(id);
}
(engine, ids)
}

fn bench_scheduler_drain(c: &mut Criterion) {
let mut group = c.benchmark_group("scheduler_drain");
for &n in &[10usize, 100, 1_000] {
// Throughput: number of rule applications in this run (n entities).
group.throughput(Throughput::Elements(n as u64));
group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
b.iter_batched(
|| build_engine_with_entities(n),
|(mut engine, ids)| {
// Apply the no-op rule to all entities, then commit.
let tx = engine.begin();
for id in &ids {
let res = engine
.apply(tx, BENCH_NOOP_RULE_NAME, id)
.expect("Failed to apply noop bench rule");
// Avoid affecting timing; check only in debug builds.
debug_assert!(matches!(res, ApplyResult::Applied));
}
let snap = engine.commit(tx).expect("Failed to commit benchmark tx");
// Ensure the commit work is not optimized away.
criterion::black_box(snap);
},
BatchSize::PerIteration,
)
});
}
group.finish();
}

criterion_group!(benches, bench_scheduler_drain);
criterion_main!(benches);
93 changes: 93 additions & 0 deletions crates/rmg-benches/benches/snapshot_hash.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#![allow(missing_docs)]
//! Benchmark: snapshot hash over a linear chain graph
//!
//! Builds a chain of `n` entities reachable from a single root node and
//! measures the cost of computing the snapshot (state_root) hash over the
//! reachable subgraph. Sizes (10, 100, 1000) provide an order-of-magnitude
//! progression to observe scaling trends without long runtimes.
//!
//! Throughput "elements" are the number of nodes in the reachable set
//! (n entities + 1 root).
//!
//! TODO(PR-14/15): Persist JSON artifacts and add a regression gate.
use criterion::{criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput};
use rmg_core::{
make_edge_id, make_node_id, make_type_id, EdgeRecord, Engine, GraphStore, NodeRecord,
};

// String constants to avoid magic literals drifting silently.
const ROOT_ID_STR: &str = "root";
const WORLD_TYPE_STR: &str = "world";
const ENTITY_TYPE_STR: &str = "entity";
const LINK_TYPE_STR: &str = "link";
const ENT_LABEL_PREFIX: &str = "ent-";

fn build_chain_engine(n: usize) -> Engine {
let mut store = GraphStore::default();
let root = make_node_id(ROOT_ID_STR);
let world = make_type_id(WORLD_TYPE_STR);
store.insert_node(
root,
NodeRecord {
ty: world,
payload: None,
},
);
// Insert N nodes and connect them in a chain so all are reachable.
let entity_ty = make_type_id(ENTITY_TYPE_STR);
let link_ty = make_type_id(LINK_TYPE_STR);
let mut chain_tail = root;
for i in 0..n {
let to_label = format!("{}{}", ENT_LABEL_PREFIX, i);
let id = make_node_id(&to_label);
store.insert_node(
id,
NodeRecord {
ty: entity_ty,
payload: None,
},
);
// Human-friendly edge id: <from>-to-<to>.
let from_label = if i == 0 {
ROOT_ID_STR.to_string()
} else {
format!("{}{}", ENT_LABEL_PREFIX, i - 1)
};
let edge_id = make_edge_id(&format!("edge-{}-to-{}", from_label, to_label));
store.insert_edge(
chain_tail,
EdgeRecord {
id: edge_id,
from: chain_tail,
to: id,
ty: link_ty,
payload: None,
},
);
chain_tail = id;
}
Engine::new(store, root)
}

fn bench_snapshot_hash(c: &mut Criterion) {
let mut group = c.benchmark_group("snapshot_hash");
for &n in &[10usize, 100, 1_000] {
// Throughput: total nodes in reachable set (n entities + 1 root).
group.throughput(Throughput::Elements(n as u64 + 1));
group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
// Build engine in setup (not timed) and measure only hashing.
b.iter_batched(
|| build_chain_engine(n),
|engine| {
let snap = engine.snapshot();
criterion::black_box(snap.hash);
},
BatchSize::SmallInput,
)
});
}
group.finish();
}

criterion_group!(benches, bench_snapshot_hash);
criterion_main!(benches);
35 changes: 35 additions & 0 deletions docs/decision-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,3 +190,38 @@ The following entries use a heading + bullets format for richer context.
- Rationale: Keep CI quiet and align with current cargo-deny schema without weakening enforcement.
- Consequence: Same effective policy, no deprecation warnings; future license exceptions remain possible via standard cargo-deny mechanisms.
- CI Note: Use `cargo-deny >= 0.14.21` in CI (workflow/container) to avoid schema drift and deprecation surprises. Pin the action/image or the downloaded binary version accordingly.

## 2025-11-02 — PR-12: benches pin + micro-optimizations

- Context: CI cargo-deny flagged wildcard policy and benches had minor inefficiencies.
- Decision:
- Pin `blake3` in `crates/rmg-benches/Cargo.toml` to `1.8.2` (no wildcard).
- `snapshot_hash`: compute `link` type id once; label edges as `e-i-(i+1)` (no `e-0-0`).
- `scheduler_drain`: builder returns `Vec<NodeId>`; `apply` loop uses precomputed ids to avoid re-hashing.
- Rationale: Keep dependency policy strict and make benches reflect best practices (no redundant hashing or id recomputation).
- Consequence: Cleaner dependency audit and slightly leaner bench setup without affecting runtime code.

## 2025-11-02 — PR-12: benches constants + documentation

- Context: Pedantic review flagged magic strings, ambiguous labels, and unclear throughput semantics in benches.
- Decision: Extract constants for ids/types; clarify edge ids as `<from>-to-<to>`; switch `snapshot_hash` to `iter_batched`; add module-level docs and comments on throughput and BatchSize; replace exact blake3 patch pin with minor pin `1.8` and document rationale.
- Rationale: Improve maintainability and readability of performance documentation while keeping timings representative.
- Consequence: Benches read as executable docs; CI docs guard updated accordingly.

## 2025-11-02 — PR-12: benches README + main link

- Context: Missing documentation for how to run/interpret Criterion benches.
- Decision: Add `crates/rmg-benches/benches/README.md` and link from the top-level `README.md`.
- Rationale: Improve discoverability and ensure new contributors can reproduce measurements.
- Consequence: Docs Guard satisfied; single-source guidance for bench usage and outputs.

## 2025-11-02 — PR-12: Sync with main + merge conflict resolution

- Context: GitHub continued to show a merge conflict on PR #113 (`echo/pr-12-snapshot-bench`).
- Decision: Merge `origin/main` into the branch (merge commit; no rebase) and resolve the conflict in `crates/rmg-benches/Cargo.toml`.
- Resolution kept:
- `license = "Apache-2.0"`, `blake3 = "1"` in dev-dependencies.
- `rmg-core = { version = "0.1.0", path = "../rmg-core" }` (version-pinned path dep per cargo-deny bans).
- Bench targets: `motion_throughput`, `snapshot_hash`, `scheduler_drain`.
- Rationale: Preserve history with a merge, align benches metadata with workspace policy, and clear PR conflict status.
- Consequence: Branch synced with `main`; local hooks (fmt, clippy, tests, rustdoc) passed; CI Docs Guard satisfied via this log and execution-plan update.
Loading