bench(eval): add rule_load comparison across load entry points#122
Merged
Conversation
Adds a Criterion benchmark group `rule_load` that compares the three public rule-load APIs at 1K / 10K / 100K rules: - `Engine::add_collection` (batched, single rebuild) - `Engine::add_rules` (batched with per-rule error collection, single rebuild) - `Engine::add_rule` in a loop (now amortized O(1) per call via the incremental `RuleIndex::append_rule` and doubling-amortized `FieldBloomIndex`) Records the numbers in `BENCHMARKS.md` under a new "Rule Load Paths (0.11.x)" subsection. All three paths scale linearly in the rule count; the single-rule loop is roughly 40% slower than batched due to incremental insert overhead plus the doubling-watermark rebuilds between 1 and 100K rules, with no quadratic blowup. Pinned numbers on M4 Pro, release build: | Rules | add_collection | add_rules | add_rule loop | |--------:|---------------:|----------:|--------------:| | 1,000 | 1.15 ms | 1.17 ms | 1.64 ms | | 10,000 | 11.82 ms | 11.85 ms | 17.23 ms | | 100,000 | 121.65 ms | 122.13 ms | 166.07 ms | Run with `cargo bench -p rsigma-eval --bench eval -- rule_load`.
5 tasks
SecurityEnthusiast
pushed a commit
to SecurityEnthusiast/rsigma
that referenced
this pull request
May 17, 2026
Replaces the placeholder Unreleased section with a full release-notes draft following the format of the v0.11.0 / v0.10.0 / v0.9.0 entries. Covers every PR merged to main since v0.11.0: - Daemon and CLI observability (PR timescale#107) - tower-http access logs, per-request OTLP tracing, batch spans, source resolution spans, DLQ visibility, NATS/sink lifecycle, correlation eviction warnings, rule load diagnostics, daemon lifecycle, global `--log-format` flag. - Eval rule loading performance (PRs timescale#119, timescale#121, timescale#122, timescale#123) - batched loaders rebuild indexes once per batch via `Engine::add_rules` / `extend_compiled_rules` / `add_collection`; single-rule path amortized O(1) via `RuleIndex::append_rule` and a doubling-watermark `FieldBloomIndex`. SigmaHQ corpus (~3,120 rules) now loads in ~120 ms. - CLI command groups (PR timescale#124) - the noun-led `engine` / `rule` / `backend` / `pipeline` / `attack` grouping with the existing migration table preserved verbatim. - Test reliability (PRs timescale#115, timescale#123) - cli_daemon_http and cli_daemon_otlp E2E suites de-flaked on macOS under load; eval bloom test made deterministic against random AHash seeds. - Dependency and CI bumps. All command-name references within the draft already use the new noun-led paths (`engine eval`, `rule validate`, etc.) so the next release ships with consistent terminology throughout the notes.
SecurityEnthusiast
pushed a commit
to SecurityEnthusiast/rsigma
that referenced
this pull request
May 20, 2026
The "operability, performance, and documentation" release. * Workspace bumped 0.11.0 -> 0.12.0; all 10 inter-crate dep pins refreshed; Cargo.lock regenerated under --locked. * CHANGELOG.md [Unreleased] section flipped to [0.12.0] - 2026-05-19; comparison link updated to v0.11.0...v0.12.0; tag reference added to the bottom-of-file link block. * CHANGELOG also gained a Documentation site (PR timescale#129) section under the existing observability / eval-perf / CLI-groups / test-reliability / dependencies headings, and the TL;DR theme moved from "operations and load performance" to "operability, performance, and documentation" to reflect the new docs site as a top-line deliverable. Covers all 13 PRs merged since v0.11.0: timescale#107 (observability), timescale#111/timescale#113/timescale#114/timescale#120 (dependency batches), timescale#115/timescale#123 (test reliability), timescale#119/timescale#121/timescale#122/timescale#123 (eval rule loading perf), timescale#124 (CLI command groups), timescale#127 (CLI docs followup), timescale#129 (documentation site).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Criterion benchmark group
rule_loadthat compares the three rule-load entry points onEngineat 1K / 10K / 100K rules and records the numbers inBENCHMARKS.mdunder a new "Rule Load Paths (0.11.x)" subsection.The benchmark exists to give empirical confirmation that all three paths scale linearly in the rule count after PR #121 made the single-rule
add_ruleamortized O(1). The existing functional regression test (test_add_rule_loop_scales_linearly_on_large_corpus) already guards against quadratic reintroduction; this PR is the matching numerical evidence.Results
M4 Pro, release build, 2026-05-16. Median time from Criterion, 10 samples per benchmark.
add_collectionadd_rulesadd_ruleloopPer-rule cost stays flat from 1K to 100K across all three paths, confirming O(N) total complexity. The single-rule loop costs roughly 40% more per rule than the batched paths, which is the amortized overhead of the incremental insert plus the doubling-watermark bloom rebuilds along the way. No quadratic blowup.
Changes
bench_rule_loadgroup incrates/rsigma-eval/benches/eval.rs, wired into bothdaachorse-indexand defaultcriterion_group!macros.BENCHMARKS.md, inserted between "Rule Compilation" and "Single Event Evaluation".Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo bench -p rsigma-eval --bench eval -- rule_load