Apply pipelines to correlation rules also

[Mat0vu]

Hi,

I´m currently testing correlation rules (using the latest release of pysigma 0.11.5 and ES|QL target from the Elastic-Backend-Repo) and realised that processing pipelines are only applied to the "regular" Sigma rule containing the detection and not to the correlation rule.

For example, I have the following pipeline which contains FieldMappingTransformations to change various field-names:

ProcessingPipeline(
        name="Test-Mapping",
        priority=60,
        allowed_backends=("elasticsearch", "esql"),
        items=
            [
            ProcessingItem(     
                identifier="windows_field_mapping",
                transformation=FieldMappingTransformation(
                    {
                        "User": "evt.data.User",
                        "TargetUserName":"evt.data.TargetUserName", 
                        "TargetDomainName":"evt.data.TargetDomainName",
                    }),
                )
            ]
        )

I tested this rule from the example section for correlation rules of the sigma specification:

title: Multiple failed logons for a single user (possible brute force attack)
status: test
id: b58f4f32-2188-4111-a708-a8c7470159ea
name: multiple_failed_logons
correlation:
    type: event_count
    rules:
        - failed_logon
    group-by:
        - TargetUserName
        - TargetDomainName
    timespan: 5m
    condition:
        gte: 10
---
title: Failed logon
name: failed_logon
status: test
id: 38d95135-c7fb-49f9-a0d8-6c4d345e1e8c
logsource:
    product: windows
    service: security
detection:
    selection:
        EventID: 4625
        User: ABC # Added to demonstrate working pipeline for regular rule
    condition: selection

After the translation with sigma convert failed_logons_correlation.yaml -t esql -p wit_winlog_mapping I get following query:

from * | where EventID==4625 and evt.data.User=="ABC"
| eval timebucket=date_trunc(5minutes, @timestamp) | stats event_count=count() by timebucket, TargetUserName, TargetDomainName
| where event_count >= 10`

This shows that User was correctly updated to evt.data.User, however TargetUserName and TargetDomainName which are used in the correlation rule were not updated.

I think this issue is because in contrast to the method convert_rule where the pipeline is processed with self.last_processing_pipeline.apply(rule) (https://github.com/SigmaHQ/pySigma/blob/main/sigma/conversion/base.py#L177)

In the method convert_correlation_rule the pipeline is not applied: https://github.com/SigmaHQ/pySigma/blob/main/sigma/conversion/base.py#L527

Because I don´t think that this behaviour is intentational and the issue is easy to fix, I added the line self.last_processing_pipeline.apply(rule) to the convert_correlation_rule for this Pull Request.

With this change, the fieldnames were correctly updated as expected, as I got this query:

from * | where EventID==4625 and evt.data.User=="ABC"
| eval timebucket=date_trunc(5minutes, @timestamp) | stats event_count=count() by timebucket, evt.data.TargetUserName, evt.data.TargetDomainName
| where event_count >= 10

Feel free to comment and change :)

[thomaspatzke]

Bugs that come with fixes are the best 😉 Thanks a lot!

I'm afraid I didn't wrote any test for this, just added some that show the failure as well as your fix.