Skip to content

feat: implement pySigma parity gaps#42

Merged
mostafa merged 3 commits into
mainfrom
fix/pysigma-parity
Apr 27, 2026
Merged

feat: implement pySigma parity gaps#42
mostafa merged 3 commits into
mainfrom
fix/pysigma-parity

Conversation

@mostafa
Copy link
Copy Markdown
Member

@mostafa mostafa commented Apr 27, 2026

Summary

Close all 8 pySigma parity gaps in correlation and filter rule handling. The work is split into two phases.

Phase 0: Structural additions (no behavioral change)

  • Add Correlation variant to RelationType enum and linter's VALID_RELATED_TYPES
  • Add related, license, fields, scope to CorrelationRule struct, parser, and KNOWN_KEYS_CORRELATION
  • Add 10 missing SigmaRuleBase fields to FilterRule with standard_filter_keys exclusion from custom_attributes
  • Widen CorrelationCondition::Threshold.field from Option<String> to Option<Vec<String>>, propagated through parser, CompiledCondition, correlation engine, pipeline field mapping, and Postgres backend
  • Add percentile: Option<u64> to CorrelationCondition::Threshold, used in ValuePercentile eval and Postgres PERCENTILE_CONT() generation

Phase 1: Behavioral fixes

  • Introduce FilterRuleTarget enum (Any/Specific). rules: any, rules: [], and omitted rules all map to Any
  • Replace symmetric logsource_compatible with asymmetric filter_logsource_contains for filter-to-rule matching
  • Use the filter's own condition expression via rewrite_condition_identifiers instead of hardcoding AND NOT. Exclusion filters now require condition: not selection in YAML, matching pySigma semantics

Lint fixup: Update filter.rules validation to accept all valid forms (omitted, empty, "any", single string, sequence) and only error on invalid types.

Breaking change

Filter rules that previously used condition: selection for exclusion now need condition: not selection. The old behavior silently hardcoded AND NOT regardless of the filter's condition expression. This was a bug: it ignored the filter's actual condition and prevented inclusion filters or multi-selection OR conditions from working.

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace (all 393 tests pass, including CLI integration tests)
  • Sigma corpus regression: rsigma validate on SigmaHQ/sigma full ruleset
  • Spot-check Postgres backend golden tests for correlation condition changes

mostafa added 3 commits April 27, 2026 23:30
Add Correlation variant to RelationType, widen condition.field to
accept arrays, add percentile to CorrelationCondition::Threshold,
and populate missing metadata fields on CorrelationRule and FilterRule.
Update linter known-keys, parser extraction, compiled condition
propagation, correlation engine field access, and Postgres backend
SQL generation to match.

Gaps addressed: 1 (multi-field condition), 2 (percentile),
3 (correlation metadata), 4 (filter metadata), 8 (RelationType).
Introduce FilterRuleTarget enum (Any vs Specific) so filters with
`rules: any` or an omitted rules field apply to every loaded rule,
matching pySigma semantics.

Replace symmetric logsource_compatible with asymmetric
filter_logsource_contains so a filter only applies when the rule
satisfies every field the filter specifies (filter fields are
constraints, not optional matches).

Use the filter's own condition expression instead of hardcoding
AND NOT. A rewrite_condition_identifiers helper namespaces all
identifier references before injection. Exclusion filters now
require `condition: not selection` in the YAML, matching pySigma.

Gaps addressed: 5 (rules: any), 6 (filter condition), 7 (logsource
containment). All existing tests updated to the new semantics.
Replace the MissingFilterRules error and EmptyFilterRules warning with
validation that accepts the newly valid forms: omitted (defaults to
"any"), empty sequence (same), and the string "any". Only error when
the value is an invalid type (e.g. a number or boolean).

Add tests for all valid forms and the invalid-type case.
@mostafa mostafa changed the title feat: close all 8 pySigma parity gaps feat: implement pySigma parity gaps Apr 27, 2026
@mostafa mostafa merged commit e600ca2 into main Apr 27, 2026
8 checks passed
@mostafa mostafa deleted the fix/pysigma-parity branch April 27, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant