[Security Solution][Detection Engine] adds alert suppression to ES|QL rule type

[vitaliidm]

Summary

• addresses https://github.com/elastic/security-team/issues/9203
• adds alert suppression for new terms rule type
• similarly to custom investigation fields list of available suppression fields:
  • shows only ES|QL fields returned in query for aggregating queries
  • shows ES|QL fields returned in query + index fields for non-aggregating queries. Since resulted alerts for this type of query, are enriched with source documents.

Demo

1. run esql rule w/o suppression
2. run esql rule w/ suppression per rule execution. Since ES|QL query is aggregating, no alerts suppressed on already agrregated field host.ip
3. run suppression on interval 20m
4. run suppression for custom ES|QL field which is the same as host.ip, hence same results
5. run suppression on interval 100m

_esql.suppression.mov

Limitations

Since suppressed alerts deduplication relies on alert timestamps, sorting of results other than @timestamp asc in ES|QL query may impact on number of suppressed alerts, when number of possible alerts more than max_signals.
This affects only non-aggregating queries, since suppression boundaries for these alerts set as rule execution time

Checklist

• Functional changes are hidden behind a feature flag

  Feature flag alertSuppressionForEsqlRuleEnabled

• Functional changes are covered with a test plan and automated tests.

  • https://github.com/elastic/security-team/pull/9389

• Stability of new and changed tests is verified using the Flaky Test Runner.

  • FTR(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5907
  • Cypress(x100): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5884

• Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless.

• Mapping changes are accompanied by a technical design document. It can be a GitHub issue or an RFC explaining the changes. The design document is shared with and approved by the appropriate teams and individual stakeholders.

  Existing AlertSuppression schema field is used for ES|QL rule, the one that already used for Query, New terms and IM rules.

      alert_suppression:
        $ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression'

  where

      AlertSuppression:
        type: object
        properties:
          group_by:
            $ref: '#/components/schemas/AlertSuppressionGroupBy'
          duration:
            $ref: '#/components/schemas/AlertSuppressionDuration'
          missing_fields_strategy:
            $ref: '#/components/schemas/AlertSuppressionMissingFieldsStrategy'
        required:
          - group_by

• Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both).

  • [Request] add ES|QL rule alert suppression docs  security-docs#5156

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[nkhristinin]

Can we add comment here and describe why we need to use hash, and not just provide unique id?

[vitaliidm]

0cef907

[nkhristinin]

Hey, thanks for you work! I did test it locally and have some questions.

I do have this index:

{
  "my-index": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "host": {
          "properties": {
            "name": {
              "type": "keyword"
            }
          }
        },
        "user": {
          "properties": {
            "name": {
              "type": "long"
            }
          }
        }
      }
    }
  }
}

this what I have without suppression:

suppression with host.name works as expected.

Question 1
I think we already discussed it, but want to just double check. It's ok, than in user.name we do have some "random" values?

Question 2

When I make change query a bit, suppression works - but there some not unique values still.

Question 3

There some duplication when you choose a field

Screen.Recording.2024-05-14.at.12.00.05.mov

[vitaliidm]

@nkhristinin

Question 1
I think we already discussed it, but want to just double check. It's ok, than in user.name we do have some "random" values?

Yes, this is expected as only one alert is created. The rest are supppressed. So, the value you see is the value from event, which is used to create alert.
Partly it is covered in https://www.elastic.co/guide/en/security/8.13/alert-suppression.html,

Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule’s criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group.

So, these "random" values are just values from the only created alert.

Question 2
When I make change query a bit, suppression works - but there some not unique values still.

You suppression configuration is to suppress per rule execution only. That means, at least one alert per group would be created during the rule execution. By default you would have 12(5m interval) rule execution in preview, so if your events appear in different executions, an alert for each one would be created.
You can switch suppression to Suppress on time interval, increase it to cover multiple rule executions(let's say 30m) and you should not see any alerts with the same values on that interval (30m)

[vitaliidm]

@nkhristinin

Question 3
There some duplication when you choose a field

Duplicate fields removed in 131929e

[MadameSheema]

sec-eng-prod changes LGTM!

[yctercero]

Should we log this error somewhere? Say a user thought their rule is correct, would they ever be alerted that something could be going wrong at this step? It'd likely error if there was some kind of syntax error, right?

[vitaliidm]

Even if there is a syntax error, AST helper should still return valid AST tree.
But there is a particular case when it crashes - typing [ after index.

The error is shown already in console.

Also reported it way back for Kibana team: #182012

[kibana-ci]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 07188d9
• Interpreting CI Failures

Failed CI Steps

• Defend Workflows Cypress Tests #2
• Defend Workflows Cypress Tests #4
• Defend Workflows Cypress Tests #5
• Defend Workflows Cypress Tests #13
• Defend Workflows Cypress Tests #17
• Defend Workflows Cypress Tests on Serverless #4

History

• 💚 Build #210087 succeeded b61f4b9
• 💚 Build #209837 succeeded 131929e
• 💚 Build #209432 succeeded b6f437a
• 💚 Build #208904 succeeded afa01ac

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm