Skip to content

feat: reverse converter (rule reverse --from <dialect>) with a Lucene frontend - #371

Merged
mostafa merged 13 commits into
mainfrom
feat/reverse-converter
Jul 21, 2026
Merged

feat: reverse converter (rule reverse --from <dialect>) with a Lucene frontend#371
mostafa merged 13 commits into
mainfrom
feat/reverse-converter

Conversation

@mostafa

@mostafa mostafa commented Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Adds a pluggable reverse-conversion framework, the mirror of the forward Backend engine: a SIEM query is parsed into the intermediate representation, raised to a Sigma rule, and emitted as YAML. Elastic Lucene ships as the reference frontend.

query -> Frontend::parse() -> IrRule -> raise_rule() -> SigmaRule -> emit_rule_yaml() -> Sigma YAML

Frontend is the inverse of Backend, raise_rule the inverse of lower_rule, and emit_rule_yaml the inverse of parse_sigma_yaml.

What's included

  • rsigma-parser — emit_rule_yaml / emit_collection_yaml, a deterministic canonical Sigma YAML emitter (sorted named detections, logsource custom fields, and custom attributes; reconstructed field|modifier keys; wildcard-escaped values with raw re/cidr/fieldref; empty logsource emitted as {}).
  • rsigma-ir — raise_rule (IrRule to SigmaRule): reconstructs the modifiers each IrMatcher implies, collapses homogeneous value lists, keeps conditions selector-preserving, and rejects numeric dynamic-source references. The canonical ir_pattern_to_sigma moved here (re-exported from rsigma-convert).
  • rsigma-convert — a reverse module: a Frontend trait plus a QueryDialect table drive a shared tokenizer and precedence-climbing boolean parser; assemble_rule builds named selections and a condition (AND-merged selections, same-field OR value lists, negated branches as filters); reverse_collection converts a batch. LuceneFrontend parses the Lucene query_string subset (field:value with wildcards, quoted phrases, /regex/, [a TO b]/{a TO b} ranges, comparison shorthand, field:(a OR b) groups, _exists_, keywords, and AND/OR/NOT with grouping) and rejects boosting, fuzzy/proximity, and non-numeric ranges with a structured error.
  • rsigma CLI — rsigma rule reverse --from <dialect>: a single entry point that selects the dialect (mirroring backend convert --target), so the rule noun does not sprout a subcommand per dialect. Reads a query from an argument, --file, or stdin; takes --title/--id/--level/--status and --logsource-* hints; prints Sigma YAML (or -o); and parses the result back before printing so a rule that would not round-trip never reaches the operator.
  • rsigma-mcp — a reverse_convert tool with a dialect parameter (13 tools total).

Reverse conversion is best-effort by design: a query carries no rule metadata, so the output is a reviewable skeleton, and inexpressible constructs are rejected rather than emitted as silently-wrong Sigma. Adding a new target is a QueryDialect table plus a Frontend::parse_atom (and a --from / dialect value).

Tests

  • Emitter and raise round-trip unit tests (parse -> lower -> raise -> emit -> parse -> lower preserves the HIR).
  • A Lucene golden corpus (crates/rsigma-convert/tests/golden/lucene/) driven through reverse_collection.
  • A fuzz_lucene_frontend target exercising the full untrusted path.
  • Full workspace: cargo fmt --check, cargo clippy --workspace --all-targets --all-features -D warnings, cargo doc (warnings denied), cargo test --workspace --all-features (3480 tests), and docmd build/validate all pass.

Closes #348.

mostafa added 10 commits July 21, 2026 17:46
@mostafa
feat(parser): add canonical Sigma YAML emitter
48eda4c 
Add emit_rule_yaml / emit_collection_yaml as the inverse of parse_sigma_yaml.
The emitter is a deterministic canonical form (sorted named detections,
logsource custom fields, and custom attributes), reconstructs field|modifier
keys, escapes literal wildcards in values while leaving re/cidr/fieldref values
raw, and reuses the ConditionExpr rendering for the condition string.
@mostafa
feat(ir): raise HIR back to a Sigma rule
28f11d8 
Add raise_rule as the inverse of lower_rule: reconstruct the field|modifier
surface each IrMatcher variant implies, collapse homogeneous value lists, keep
conditions selector-preserving, and reject numeric dynamic-source references.
Move the canonical ir_pattern_to_sigma into rsigma-ir (re-exported from
rsigma-convert) so convert and the raise path share one reconstruction.

Round-trip tested: parse -> lower -> raise -> lower is HIR-stable, and the full
parse -> lower -> raise -> emit -> parse -> lower pivot preserves the HIR.
@mostafa
feat(convert): reverse conversion framework with a Lucene frontend
1b0d1d4 
Add rsigma-convert::reverse, the mirror of the Backend engine: a Frontend trait
plus a QueryDialect table drive a shared tokenizer and precedence-climbing
boolean parser, and assemble a boolean tree of leaves into named Sigma
selections and a condition (AND-merged selections, same-field OR value lists,
negated branches as filters). raise_rule and emit_rule_yaml turn the resulting
IrRule into Sigma YAML; reverse_collection converts a batch, collecting
per-query errors.

The Lucene reference frontend parses the query_string subset detection authors
use (field:value with wildcards, quoted phrases, /regex/, [a TO b] ranges,
comparison shorthand, field:(a OR b) groups, _exists_, keywords, AND/OR/NOT with
grouping) and rejects boosting, fuzzy/proximity, and non-numeric ranges with a
structured ConvertError.
@mostafa
fix(parser): emit an empty logsource as {} so it re-parses
3135231 
A rule with no logsource fields previously emitted a bare `logsource:` key,
which parses as null and the parser rejects (logsource must be a mapping). Emit
an explicit empty mapping instead.
@mostafa
feat(cli): add rsigma rule from-lucene
6d56c24 
Convert an Elastic Lucene query into a draft Sigma rule. Reads the query from
an argument, --file, or stdin; takes title/id/level/status/logsource hints a
query cannot carry; and prints Sigma YAML (or writes it with -o). The emitted
rule is parsed back before printing, so a rule that would not round-trip never
reaches the operator. Joins the rule draft / discover-schemas authoring family.
@mostafa
feat(mcp): add the from_lucene tool
fbe24bc 
Expose reverse conversion over MCP: from_lucene turns an Elastic Lucene query
into a draft Sigma rule (YAML), taking the metadata and logsource a query
cannot carry as parameters and returning inexpressible constructs as an error
envelope. Brings the tool count to 13.
@mostafa
test(convert): Lucene reverse goldens and a fuzz target
37cd5d5 
Add a golden corpus (query -> expected Sigma YAML) covering equality, AND/NOT
filters, same-field OR value lists, regex, ranges and comparisons, value
groups, _exists_, and keywords, driven through reverse_collection like the CLI.
Add the fuzz_lucene_frontend target exercising the full untrusted path
(tokenize, parse, assemble, raise, emit). Also fix quoted-phrase unescaping so
a Lucene \\ inside quotes becomes a single literal backslash.
@mostafa
docs: document reverse conversion and rule from-lucene
9454050 
CHANGELOG entry for the reverse-conversion framework and Lucene frontend;
rsigma-parser (emit), rsigma-ir (raise), rsigma-convert (reverse module) and
rsigma-mcp (from_lucene tool) README updates; and a new rule from-lucene CLI
docs page wired into the site nav.
@mostafa
docs: drop redundant rustdoc link targets
83890f2
@mostafa
docs(changelog): reference PR #371
5934188
@mostafa
mostafa marked this pull request as draft July 21, 2026 16:38
@mostafa
refactor(cli,mcp): expose reverse conversion via a single dialect-sel…
9822da0 
…ecting entry point

Replace the per-dialect `rule from-lucene` / `from_lucene` surface with one
entry point that selects the dialect, mirroring the forward `backend convert
--target`: the CLI is now `rsigma rule reverse --from <dialect>` (a --from value
enum) and the MCP tool is `reverse_convert` with a `dialect` parameter. This
keeps the rule noun from sprouting a subcommand per dialect and scales cleanly
as SPL/KQL/others land. Lucene remains the only dialect.
@mostafa mostafa changed the title feat: reverse converter (query to Sigma YAML) with a Lucene frontend feat: reverse converter (rule reverse --from <dialect>) with a Lucene frontend Jul 21, 2026
mostafa added 2 commits July 21, 2026 18:54
@mostafa
test(mcp): cover reverse_convert in the smoke script
969ca7a 
Add the reverse_convert tool call to the mcp-smoke surface (13 tools) and update
the count in the docstring. Verified end to end over both transports.
@mostafa
feat(cli): rule reverse reads queries from files and directories
587ad09 
--file is now repeatable and accepts files or directories: each file is one
query, a directory contributes every query file it holds (recursively, filtered
to the dialect's extensions; an explicitly named file is read regardless).
Batch runs convert one rule per query, titled from the file name, and emit a
multi-document stream (or one <name>.yml per query when -o is a directory).
Unconvertible queries are reported and fail the run without dropping the rules
that did convert.
@mostafa
mostafa marked this pull request as ready for review July 21, 2026 17:11
@mostafa
mostafa merged commit cbc4deb into main Jul 21, 2026
18 checks passed
@mostafa
mostafa deleted the feat/reverse-converter branch July 21, 2026 17:23
@mostafa mostafa mentioned this pull request Jul 22, 2026
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Reverse converter (query to Sigma YAML)

1 participant

@mostafa