Add opt-in RSigma library backend for Sigma detection#101
Conversation
Run timescale/rsigma-action on rules/sigma changes to lint and validate the bundled Sigma rules and diff rule fields against the PR merge-base. The action installs a SLSA-verified rsigma release, annotates findings on the diff, and keeps a sticky summary comment on same-repo pull requests.
Pin actions/checkout and timescale/rsigma-action by full commit SHA, harden the checkout (persist-credentials: false), scope permissions to least privilege with a concurrency group, and select the rsigma v0.18.0 release. Passes zizmor --persona=pedantic with zero findings.
Add optional rsigma-parser and rsigma-eval (0.18) dependencies behind a new rsigma-engine feature. The feature is off by default and nothing consumes the crates yet, so the default build is unchanged; this is the scaffolding for the RSigma-backed Sigma engine.
Implement rsigma_eval::Event over a borrowed NormalizedEvent behind the rsigma-engine feature. Field lookups stay zero-copy by delegating to NormalizedEvent::get_field, so no per-event JSON is needed on the detection hot path. Keyword search and field enumeration are cold paths that materialize the Sigma-cased fields via serde, keeping them in sync with the field structs' rename attributes. Includes unit tests over generic and typed process events.
Wire the rsigma-engine feature to a working backend behind Rustinel's existing Engine API. Rules are parsed with rsigma-parser and compiled into one rsigma_eval::Engine per normalized logsource, reusing Rustinel's platform and logsource classification so bucketing and skip stats are unchanged. check_event routes each NormalizedEvent through the same candidate logsources, evaluates it through the zero-copy adapter, and maps the first detection to an Alert, with match-detail translation gated on the existing match-debug setting. The built-in matcher, condition transpiler, and rule types now compile only for the default backend. The full existing test suite passes under --features rsigma-engine, and both configurations are clippy- and fmt-clean.
Add a parity test that exercises cidr, re, and contains|all through the public Engine API so it runs identically under the built-in and RSigma backends; running it under both backends in CI is the parity guarantee. Add a workflow that builds and runs the full suite with --features rsigma-engine on macOS and Windows (no eBPF object needed), keeping the RSigma-backed engine in lockstep with the default one.
Add a check_event throughput benchmark over a representative Linux ruleset and mixed process/network/file/DNS events. The backend is selected at compile time, so results are labelled per backend and the built-in and RSigma engines are compared by running the benchmark with and without --features rsigma-engine.
Extend benches/sigma_backend.rs with a large synthetic ruleset (~2,000 rules) so the benchmark also exercises the regime where rule indexing matters, and add BENCHMARK.md documenting how to run the built-in vs RSigma comparison and the current results: RSigma trades a small fixed overhead on trivial rulesets for a ~1.4x throughput advantage at production rule counts.
Rework the backend seam so the built-in matcher and the RSigma engine are both compiled in when the rsigma-engine feature is enabled, and pick between them at runtime instead of replacing one with the other at compile time. Add a SigmaEngineKind selector, a --sigma-engine CLI flag on `run`, and a scanner.sigma_engine config key (EDR__SCANNER__SIGMA_ENGINE honored), defaulting to the built-in engine. Requesting rsigma without the feature fails fast at startup. Engine::check_event, load_rule, and stats dispatch on the selected backend; the runtime and hot-reload paths thread the selection through. The parity test now builds one engine per available backend in-process and asserts they agree, and the benchmark builds the feature-selected backend.
Compile the release artifacts with --features rsigma-engine so the published binaries expose the RSigma engine at runtime via --sigma-engine, while the default developer build stays built-in only.
Describe the built-in and RSigma backends and how to select them via --sigma-engine, scanner.sigma_engine, or EDR__SCANNER__SIGMA_ENGINE in the detection guide, and add the flag and config key to the CLI and configuration references.
Add an Engine Conformance section to the detection guide summarizing the Sigma features the built-in matcher does not implement (N-of quantifiers, array-scope quantifiers, correlations, filters, reset/repeat actions, expand, sigma-version, full rule metadata) and that RSigma covers them, next to the engine-selection instructions.
Link the conformance table's array-scope quantifier and sigma-version rows to SigmaHQ SEP #212 and SEP #213, and note both are proposed enhancements for the next major spec that RSigma implements as the reference implementation.
|
Thanks for the spike. I reviewed it and the direction looks good. This validates the approach from #85: RSigma is integrated as an optional stateless backend, Rustinel keeps normalization/logsource routing/ECS output/hot reload ownership, and the full I think we should include Since this means RSigma becomes a shipped opt-in backend, I suggest documenting it clearly as experimental for now. The important point is not that it is a third-party dependency, since we already ship dependencies, but that it is a new detection backend and may have behavior differences from the built-in matcher. Before merging, I would do a small finalization pass:
|
Drop the feature gate on the rsigma CLI value so `--sigma-engine rsigma` parses in every build and then fails through the same startup resolver as the config and env value, with the clear "built without rsigma-engine" message, instead of being rejected by the argument parser.
Add tests for SigmaEngineKind parsing and resolution (config value, invalid value, CLI override precedence, and rsigma requested with and without the feature), plus config tests for the builtin default and the EDR__SCANNER__SIGMA_ENGINE override.
Note in the detection guide that RSigma is an opt-in experimental backend that may differ in behavior from the built-in matcher, which stays the default.
Move the built-in vs RSigma micro-benchmark content into docs/benchmarking.md as a dedicated section and remove the top-level BENCHMARK.md so benchmarking docs live in one place.
|
Thanks for the review. Finalization pass pushed, mapping to your points:
|
Summary
Adds an opt-in RSigma library backend for Sigma detection, selectable at runtime, while keeping Rustinel-owned normalization, logsource classification, ECS alerting, hot reload, and IOC and YARA unchanged. The built-in matcher stays the default, so existing deployments are unaffected. RSigma ships as an opt-in experimental backend: it is a new detection path that may differ in behavior from the built-in matcher.
rsigma-engineCargo feature compiles the RSigma engine (rsigma-parser+rsigma-eval, v0.18) in alongside the built-in matcher; both are present in feature builds and chosen at runtime.rustinel run --sigma-engine <builtin|rsigma>, thescanner.sigma_engineconfig key, orEDR__SCANNER__SIGMA_ENGINE. The default isbuiltin.--sigma-engine rsigmaparses in every build; requestingrsigmafrom a binary built without the feature fails fast at startup through the shared resolver with a clear "built without rsigma-engine" message.rsigma_eval::Eventadapter overNormalizedEvent, so stateless detection needs no per-event JSON on the hot path.rsigma-parserand compiles into one engine per normalized logsource, reusing the existing platform and logsource load decisions so rule bucketing and skip stats match the built-in path.EvaluationResultis mapped back to Rustinel'sAlert(severity, rule id, and match-detail translation gated on the existing match-debug setting).EngineAPI is preserved;check_event,load_rule, andstatsdispatch on the selected backend, sohandler,reload, andruntimeare unchanged apart from threading the selection.--features rsigma-engineso the flag works out of the box; the default developer build is built-in only.Also included:
timescale/rsigma-actionto lint and validate the bundled Sigma rules.--features rsigma-engine; a parity test builds both backends in-process and asserts they agree.EDR__SCANNER__SIGMA_ENGINEoverride, invalid value,rsigmawith and without the feature, and CLI precedence over config.docs/benchmarking.md, comparing the two backends.Correlation (RSigma's stateful engine) is intentionally out of scope here and can follow as a separate opt-in feature.
Refs #85.
Test plan
cargo test(built-in backend)cargo test --features rsigma-engine(both backends; the parity test compares them in-process)cargo clippy --all-targets -- -D clippy::all, and the same with--features rsigma-enginecargo fmt --all -- --checkrustinel run --sigma-engine rsigmaselects the RSigma engine; the same flag on a non-feature build errors at startupcargo bench --bench sigma_backendwith and without--features rsigma-enginePerformance
On a trivial ruleset the built-in matcher is marginally faster; on a ~2,000-rule corpus the RSigma backend is about 1.4x faster. Details in
docs/benchmarking.md.