[Sigma Filters] Introducing Sigma Filters (Sigma Defeats) in Alpha / Development Preview

[sifex]

Context

This is a PR to introduce Sigma Filters into pySigma. This is the outcome of the discussions that occurred back in Dec 2022 here. SigmaHQ/sigma-specification#35

Explanation

The current implementation of Filters is as per the Sigma Specifications.

title: Filter Out Domain Controllers
description: Filter out events from Domain Controllers
logsource:
  product: windows
filter:
  rules:
    - 6f3e2987-db24-4c78-a860-b4f4095a7095 # Data Compressed - rar.exe
    - df0841c0-9846-4e9f-ad8a-7df91571771b # Login on jump host
  selection:
    ComputerName|startswith: "DC-"
  condition: not selection

Updates / Changes

This PR extends SigmaFilter from SigmaRuleBase, and is compatible with loading into SigmaCollection. apply_on_rule is the main focus of the change which applies the filter upon a given rule's condition tree.

pySigma/sigma/filters.py

Lines 166 to 189 in e230618

	def apply_on_rule(
	self, rule: Union[SigmaRule, SigmaCorrelationRule]
	) -> Union[SigmaRule, SigmaCorrelationRule]:
	if not self._should_apply_on_rule(rule):
	return rule
	for original_cond_name, condition in self.global_filter.detections.items():
	cond_name = "_filt_" + ("".join(random.choices(string.ascii_lowercase, k=10)))
	# Replace each instance of the original condition name with the new condition name to avoid conflicts
	self.global_filter.condition[0] = re.sub(
	rf"[^ ]*{original_cond_name}[^ ]*",
	cond_name,
	self.global_filter.condition[0],
	)
	rule.detection.detections[cond_name] = condition
	for i, condition in enumerate(rule.detection.condition):
	rule.detection.condition[i] = (
	f"({condition}) and " + f"({self.global_filter.condition[0]})"
	)
	# Reparse the rule to update the parsed conditions
	rule.detection.__post_init__()

---

The other major change is to collection.py, which applies a reduce function on the rules attribute,– if there are filters – otherwise leaves the rules untouched.

pySigma/sigma/collection.py

Lines 45 to 70 in e230618

	def resolve_rule_references(self):
	"""
	Resolve rule references in correlation rules to the actual rule objects and sort the rules
	by reference order (rules that are referenced by other rules come first).
	This must be called before referencing rules are converted into queries to make references available.
	"""
	for rule in self.rules:
	# Resolves all rule references in the rules property to actual Sigma rules.
	if isinstance(rule, SigmaCorrelationRule):
	rule.resolve_rule_references(self)
	# Extract all filters from the rules
	filters: List[SigmaFilter] = [rule for rule in self.rules if isinstance(rule, SigmaFilter)]
	self.rules = [rule for rule in self.rules if not isinstance(rule, SigmaFilter)]
	# Apply filters on each rule and replace the rule with the filtered rule
	self.rules = (
	[reduce(lambda r, f: f.apply_on_rule(r), filters, rule) for rule in self.rules]
	if filters
	else self.rules
	)
	# Sort rules by reference order
	self.rules = list(sorted(self.rules))

Edge Cases

I'm not 100% solid on logsource + rule IDs being the only way Filters should apply, but certainly a warning should be shown in the following scenarios:

• Rules aren't specified in the Filter YAML, in which case the Filter will apply if the logsource matches or is a subset.
• Logsource is omitted / missing (probably an error)

Filtering In vs Filtering Out

In design, I currently implemented filtering-in by default (AND) and not filtering-out (NOT) as the condition field should really be used for this. imho this is the expected approach to filtering when using these files first hand.

   condition: selection # filtering in / AND
---
   condition: not selection # filtering out / NOT

Checklist

• Adds test coverage to cover the main elements of the feature set.

[frack113]

Why not add a generic SigmaGlobalFilterError Class like the SigmaCollectionError ?

[sifex]

@thomaspatzke let me know if there's any changes and if this is all good on your end ☺️

[thomaspatzke]

Very cool! Please check my comments.

[sifex]

Resolved all I can, should be 2 left to respond to ☺️

[thomaspatzke]

Oops, I clicked on resolve all by accident. I unresolved two but unsure. Please check if these were the right ones.