feat: implement pySigma parity gaps#42
Merged
Merged
Conversation
Add Correlation variant to RelationType, widen condition.field to accept arrays, add percentile to CorrelationCondition::Threshold, and populate missing metadata fields on CorrelationRule and FilterRule. Update linter known-keys, parser extraction, compiled condition propagation, correlation engine field access, and Postgres backend SQL generation to match. Gaps addressed: 1 (multi-field condition), 2 (percentile), 3 (correlation metadata), 4 (filter metadata), 8 (RelationType).
Introduce FilterRuleTarget enum (Any vs Specific) so filters with `rules: any` or an omitted rules field apply to every loaded rule, matching pySigma semantics. Replace symmetric logsource_compatible with asymmetric filter_logsource_contains so a filter only applies when the rule satisfies every field the filter specifies (filter fields are constraints, not optional matches). Use the filter's own condition expression instead of hardcoding AND NOT. A rewrite_condition_identifiers helper namespaces all identifier references before injection. Exclusion filters now require `condition: not selection` in the YAML, matching pySigma. Gaps addressed: 5 (rules: any), 6 (filter condition), 7 (logsource containment). All existing tests updated to the new semantics.
Replace the MissingFilterRules error and EmptyFilterRules warning with validation that accepts the newly valid forms: omitted (defaults to "any"), empty sequence (same), and the string "any". Only error when the value is an invalid type (e.g. a number or boolean). Add tests for all valid forms and the invalid-type case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Close all 8 pySigma parity gaps in correlation and filter rule handling. The work is split into two phases.
Phase 0: Structural additions (no behavioral change)
Correlationvariant toRelationTypeenum and linter'sVALID_RELATED_TYPESrelated,license,fields,scopetoCorrelationRulestruct, parser, andKNOWN_KEYS_CORRELATIONSigmaRuleBasefields toFilterRulewithstandard_filter_keysexclusion fromcustom_attributesCorrelationCondition::Threshold.fieldfromOption<String>toOption<Vec<String>>, propagated through parser,CompiledCondition, correlation engine, pipeline field mapping, and Postgres backendpercentile: Option<u64>toCorrelationCondition::Threshold, used inValuePercentileeval and PostgresPERCENTILE_CONT()generationPhase 1: Behavioral fixes
FilterRuleTargetenum (Any/Specific).rules: any,rules: [], and omittedrulesall map toAnylogsource_compatiblewith asymmetricfilter_logsource_containsfor filter-to-rule matchingrewrite_condition_identifiersinstead of hardcodingAND NOT. Exclusion filters now requirecondition: not selectionin YAML, matching pySigma semanticsLint fixup: Update
filter.rulesvalidation to accept all valid forms (omitted, empty,"any", single string, sequence) and only error on invalid types.Breaking change
Filter rules that previously used
condition: selectionfor exclusion now needcondition: not selection. The old behavior silently hardcodedAND NOTregardless of the filter's condition expression. This was a bug: it ignored the filter's actual condition and prevented inclusion filters or multi-selection OR conditions from working.Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features -- -D warningscargo test --workspace(all 393 tests pass, including CLI integration tests)rsigma validateon SigmaHQ/sigma full ruleset