Skip to content

Add opt-in RSigma library backend for Sigma detection#101

Merged
Karib0u merged 17 commits into
Karib0u:mainfrom
mostafa:feat/rsigma-engine
Jul 2, 2026
Merged

Add opt-in RSigma library backend for Sigma detection#101
Karib0u merged 17 commits into
Karib0u:mainfrom
mostafa:feat/rsigma-engine

Conversation

@mostafa

@mostafa mostafa commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in RSigma library backend for Sigma detection, selectable at runtime, while keeping Rustinel-owned normalization, logsource classification, ECS alerting, hot reload, and IOC and YARA unchanged. The built-in matcher stays the default, so existing deployments are unaffected. RSigma ships as an opt-in experimental backend: it is a new detection path that may differ in behavior from the built-in matcher.

  • The rsigma-engine Cargo feature compiles the RSigma engine (rsigma-parser + rsigma-eval, v0.18) in alongside the built-in matcher; both are present in feature builds and chosen at runtime.
  • Select the engine with rustinel run --sigma-engine <builtin|rsigma>, the scanner.sigma_engine config key, or EDR__SCANNER__SIGMA_ENGINE. The default is builtin. --sigma-engine rsigma parses in every build; requesting rsigma from a binary built without the feature fails fast at startup through the shared resolver with a clear "built without rsigma-engine" message.
  • Zero-copy rsigma_eval::Event adapter over NormalizedEvent, so stateless detection needs no per-event JSON on the hot path.
  • The RSigma loader parses with rsigma-parser and compiles into one engine per normalized logsource, reusing the existing platform and logsource load decisions so rule bucketing and skip stats match the built-in path. EvaluationResult is mapped back to Rustinel's Alert (severity, rule id, and match-detail translation gated on the existing match-debug setting).
  • The public Engine API is preserved; check_event, load_rule, and stats dispatch on the selected backend, so handler, reload, and runtime are unchanged apart from threading the selection.
  • The official release binaries are built with --features rsigma-engine so the flag works out of the box; the default developer build is built-in only.

Also included:

  • A Detection-as-Code CI gate using timescale/rsigma-action to lint and validate the bundled Sigma rules.
  • A workflow that runs the full suite under --features rsigma-engine; a parity test builds both backends in-process and asserts they agree.
  • Engine-selection resolution tests: config value, EDR__SCANNER__SIGMA_ENGINE override, invalid value, rsigma with and without the feature, and CLI precedence over config.
  • A Criterion benchmark documented in docs/benchmarking.md, comparing the two backends.
  • Docs for engine selection in the detection guide and the CLI and configuration references, including an engine-conformance table and the experimental note.

Correlation (RSigma's stateful engine) is intentionally out of scope here and can follow as a separate opt-in feature.

Refs #85.

Test plan

  • cargo test (built-in backend)
  • cargo test --features rsigma-engine (both backends; the parity test compares them in-process)
  • cargo clippy --all-targets -- -D clippy::all, and the same with --features rsigma-engine
  • cargo fmt --all -- --check
  • rustinel run --sigma-engine rsigma selects the RSigma engine; the same flag on a non-feature build errors at startup
  • cargo bench --bench sigma_backend with and without --features rsigma-engine

Performance

On a trivial ruleset the built-in matcher is marginally faster; on a ~2,000-rule corpus the RSigma backend is about 1.4x faster. Details in docs/benchmarking.md.

mostafa added 8 commits July 2, 2026 09:38
Run timescale/rsigma-action on rules/sigma changes to lint and validate the
bundled Sigma rules and diff rule fields against the PR merge-base. The action
installs a SLSA-verified rsigma release, annotates findings on the diff, and
keeps a sticky summary comment on same-repo pull requests.
Pin actions/checkout and timescale/rsigma-action by full commit SHA, harden
the checkout (persist-credentials: false), scope permissions to least
privilege with a concurrency group, and select the rsigma v0.18.0 release.
Passes zizmor --persona=pedantic with zero findings.
Add optional rsigma-parser and rsigma-eval (0.18) dependencies behind a new
rsigma-engine feature. The feature is off by default and nothing consumes the
crates yet, so the default build is unchanged; this is the scaffolding for the
RSigma-backed Sigma engine.
Implement rsigma_eval::Event over a borrowed NormalizedEvent behind the
rsigma-engine feature. Field lookups stay zero-copy by delegating to
NormalizedEvent::get_field, so no per-event JSON is needed on the detection
hot path. Keyword search and field enumeration are cold paths that materialize
the Sigma-cased fields via serde, keeping them in sync with the field structs'
rename attributes. Includes unit tests over generic and typed process events.
Wire the rsigma-engine feature to a working backend behind Rustinel's existing
Engine API. Rules are parsed with rsigma-parser and compiled into one
rsigma_eval::Engine per normalized logsource, reusing Rustinel's platform and
logsource classification so bucketing and skip stats are unchanged. check_event
routes each NormalizedEvent through the same candidate logsources, evaluates it
through the zero-copy adapter, and maps the first detection to an Alert, with
match-detail translation gated on the existing match-debug setting.

The built-in matcher, condition transpiler, and rule types now compile only for
the default backend. The full existing test suite passes under
--features rsigma-engine, and both configurations are clippy- and fmt-clean.
Add a parity test that exercises cidr, re, and contains|all through the public
Engine API so it runs identically under the built-in and RSigma backends;
running it under both backends in CI is the parity guarantee. Add a workflow
that builds and runs the full suite with --features rsigma-engine on macOS and
Windows (no eBPF object needed), keeping the RSigma-backed engine in lockstep
with the default one.
Add a check_event throughput benchmark over a representative Linux ruleset and
mixed process/network/file/DNS events. The backend is selected at compile time,
so results are labelled per backend and the built-in and RSigma engines are
compared by running the benchmark with and without --features rsigma-engine.
Extend benches/sigma_backend.rs with a large synthetic ruleset (~2,000 rules)
so the benchmark also exercises the regime where rule indexing matters, and add
BENCHMARK.md documenting how to run the built-in vs RSigma comparison and the
current results: RSigma trades a small fixed overhead on trivial rulesets for a
~1.4x throughput advantage at production rule counts.
@mostafa mostafa marked this pull request as draft July 2, 2026 09:15
mostafa added 3 commits July 2, 2026 12:05
Rework the backend seam so the built-in matcher and the RSigma engine are both
compiled in when the rsigma-engine feature is enabled, and pick between them at
runtime instead of replacing one with the other at compile time. Add a
SigmaEngineKind selector, a --sigma-engine CLI flag on `run`, and a
scanner.sigma_engine config key (EDR__SCANNER__SIGMA_ENGINE honored), defaulting
to the built-in engine. Requesting rsigma without the feature fails fast at
startup. Engine::check_event, load_rule, and stats dispatch on the selected
backend; the runtime and hot-reload paths thread the selection through.

The parity test now builds one engine per available backend in-process and
asserts they agree, and the benchmark builds the feature-selected backend.
Compile the release artifacts with --features rsigma-engine so the published
binaries expose the RSigma engine at runtime via --sigma-engine, while the
default developer build stays built-in only.
Describe the built-in and RSigma backends and how to select them via
--sigma-engine, scanner.sigma_engine, or EDR__SCANNER__SIGMA_ENGINE in the
detection guide, and add the flag and config key to the CLI and configuration
references.
@mostafa mostafa marked this pull request as ready for review July 2, 2026 10:18
mostafa added 2 commits July 2, 2026 12:42
Add an Engine Conformance section to the detection guide summarizing the Sigma
features the built-in matcher does not implement (N-of quantifiers, array-scope
quantifiers, correlations, filters, reset/repeat actions, expand, sigma-version,
full rule metadata) and that RSigma covers them, next to the engine-selection
instructions.
Link the conformance table's array-scope quantifier and sigma-version rows to
SigmaHQ SEP #212 and SEP #213, and note both are proposed enhancements for the
next major spec that RSigma implements as the reference implementation.
@Karib0u

Karib0u commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Thanks for the spike. I reviewed it and the direction looks good.

This validates the approach from #85: RSigma is integrated as an optional stateless backend, Rustinel keeps normalization/logsource routing/ECS output/hot reload ownership, and the full rsigma-runtime is not introduced.

I think we should include rsigma-engine in the official release binaries. The default remains builtin, so existing deployments are unchanged, but users can try --sigma-engine rsigma without rebuilding from source. That should give us much better real-world feedback on parity, diagnostics, and performance.

Since this means RSigma becomes a shipped opt-in backend, I suggest documenting it clearly as experimental for now. The important point is not that it is a third-party dependency, since we already ship dependencies, but that it is a new detection backend and may have behavior differences from the built-in matcher.

Before merging, I would do a small finalization pass:

  • Add explicit tests for engine selection resolution:
    • config value
    • env override (EDR__SCANNER__SIGMA_ENGINE)
    • invalid engine value
    • requesting rsigma without the feature
    • CLI override precedence over config
  • Clarify non-feature CLI behavior. Ideally --sigma-engine rsigma should parse in all builds, then fail through the same startup resolver as config/env with the clear “built without rsigma-engine” message.
  • Keep release binaries built with --features rsigma-engine, but document RSigma as an opt-in experimental backend for now.
  • Remove BENCHMARK.md as a top-level file, or move/merge that content into docs/benchmarking.md so benchmarking docs stay in one place.

mostafa added 4 commits July 2, 2026 18:41
Drop the feature gate on the rsigma CLI value so `--sigma-engine rsigma` parses
in every build and then fails through the same startup resolver as the config
and env value, with the clear "built without rsigma-engine" message, instead of
being rejected by the argument parser.
Add tests for SigmaEngineKind parsing and resolution (config value, invalid
value, CLI override precedence, and rsigma requested with and without the
feature), plus config tests for the builtin default and the
EDR__SCANNER__SIGMA_ENGINE override.
Note in the detection guide that RSigma is an opt-in experimental backend that
may differ in behavior from the built-in matcher, which stays the default.
Move the built-in vs RSigma micro-benchmark content into docs/benchmarking.md as
a dedicated section and remove the top-level BENCHMARK.md so benchmarking docs
live in one place.
@mostafa

mostafa commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Finalization pass pushed, mapping to your points:

  • Engine-selection resolution tests: added unit tests over SigmaEngineKind::parse/resolve (config value, invalid value, CLI override precedence, and rsigma requested with and without the feature) plus config tests for the builtin default and the EDR__SCANNER__SIGMA_ENGINE override. (db9b79c, 49edbf5)
  • Non-feature CLI behavior: --sigma-engine rsigma now parses in every build and fails through the same startup resolver as the config and env value, with the clear "built without rsigma-engine" message, instead of being rejected by the argument parser. (db9b79c)
  • Release binaries keep --features rsigma-engine, and RSigma is now documented as an opt-in experimental backend in the detection guide, next to the engine-conformance table. (88f1529)
  • Removed the top-level BENCHMARK.md and moved its content into docs/benchmarking.md as a dedicated Sigma-engine micro-benchmark section, so benchmarking docs live in one place. (7de1c00)

cargo test, clippy, and fmt are green in both the default and --features rsigma-engine configurations.

@Karib0u Karib0u merged commit d2da031 into Karib0u:main Jul 2, 2026
16 checks passed
@mostafa mostafa deleted the feat/rsigma-engine branch July 2, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants