[Security Solution][Detection Engine] adds data tier filters to Kibana advanced settings #186908

vitaliidm · 2024-06-25T16:43:20Z

Summary

addresses https://github.com/elastic/security-team/issues/9228
introduces new Kibana advanced settings option securitySolution:excludedDataTiersForRuleExecution, that allows to exclude cold and frozen data tiers from search during rule execution
- users would be able to add data_cold or/and data_frozen tiers
ES|QL rule does not support this feature: ESQL: Allow limiting search to certain data tiers elasticsearch#108264
Machine learning rule does not support this feature
Advanced setting available only for ESS

UI

Demo

Checking there are 3M+ documents in cold data tier of test-frozen index
When rule executes, it generates alerts.
Checking kibana ancestor index of generated alert - it's restored-test-frozen-000001, which confirms alert was created from a document in cold tier
In advanced settings exlcude data_cold tier
Execute rule again, observe no alerts were created

Screen.Recording.2024-07-04.at.17.30.18.mov

How to test

Create a deployment with cold and frozen data tiers and use following commands to create index and ILM

Data tiers commands

PUT /_cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "1m"
  }
}


PUT /_ilm/policy/filtering_data_tiers
{
  "policy": {
    "phases": {
        "frozen": {
          "min_age": "10m",
          "actions": {
            "searchable_snapshot": {
              "snapshot_repository": "found-snapshots",
              "force_merge_index": true
            }
          }
        },
        "cold": {
          "min_age": "1m",
          "actions": {
            "searchable_snapshot": {
              "snapshot_repository": "found-snapshots",
              "force_merge_index": true
            },
            "set_priority": {
              "priority": 0
            }
          }
        },
        "hot": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            }
          }
        }
    }
  }
}


PUT /_index_template/filtering_data_tiers_template
{
  "index_patterns": [
    "filtering_data_tiers*"
  ],
  "template": {
    "settings": {
      "index.lifecycle.name": "filtering_data_tiers",
      "index.lifecycle.rollover_alias": "test-filtering_data_tiers"
    },
    "mappings": {
      "_meta": {
        "version": "1.6.0"
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "host": {
          "properties": {
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    }
  }
}

PUT /filtering_data_tiers-000001
{
  "aliases": {
    "filtering_data_tiers": {
      "is_write_index": true
    }
  }
}


POST filtering_data_tiers/_doc
{
  "@timestamp": "2024-07-08T17:00:01.000Z",
  "host.name": "test-0"
}

OR
reach out to @vitaliidm to get access to already existing deployment/pR deployment, where test-frozen index has cold and frozen nodes and ILM policy that move any data to a tier according to config.

Check number of documents in tier by

GET test-frozen/_count
{
    "query": {
     "bool": {
        "must": {
          "terms": {
            "_tier": ["data_cold"]
          }
        }
     }
   }
}

Create rule of supported type and query that index

Checklist

Functional changes are covered with a test plan and automated tests.
- https://github.com/elastic/security-team/pull/9896
Comprehensive manual testing is done by two engineers: the PR author and one of the PR reviewers. Changes are tested in both ESS and Serverless.
Functional changes are communicated to the Docs team. A ticket or PR is opened in https://github.com/elastic/security-docs. The following information is included: any feature flags used, affected environments (Serverless, ESS, or both).
- [Request] Document filtering out cold/frozen data tiers during rule execution security-docs#5483

…a advanced settings

…://github.com/vitaliidm/kibana into de_8_16/add_data_tiers_filtering_adv_settings

vitaliidm · 2024-07-10T14:47:03Z

x-pack/plugins/security_solution/server/ui_settings.ts

+      description: i18n.translate(
+        'xpack.securitySolution.uiSettings.excludedDataTiersForRuleExecutionDescription',
+        {
+          defaultMessage: `


@nastasha-solomon, can you have a look at this advanced settings name and description?
Let us know, if we can improved it

Thanks

@vitaliidm, what do you think about aligning this advanced setting's name with the advanced setting that excludes cold and frozen data tiers in Analyzer (i.e., change excludedDataTiersForRuleExecution to excludeColdAndFrozenTiersInRules or excludeSpecifiedTiersInRules )?

I also really like that the excludeColdAndFrozenTiersInAnalyzerdescription explains what the setting does, rather than when you'd use it. There are n number of use cases for toggling a setting that includes/excludes data in a query, so it might be more practical to inform user about what will happen if they use the setting instead of why it's useful for them. Here's a potential revision to consider:

When enabled, events from the specified data tiers are not searched during rules executions. If you specify multiple data tiers, separate values with commas. For example: data_frozen,data_cold

If you want to keep the bit about the value that the setting's offering, here's an alternative:

When enabled, events from the specified data tiers are not searched during rules executions. This might help to improve rule performance or reduce execution time. If you specify multiple data tiers, separate values with commas. For example: data_frozen,data_cold

what do you think about aligning this advanced setting's name with the advanced setting that excludes cold and frozen data tiers in Analyzer (i.e., change excludedDataTiersForRuleExecution to excludeColdAndFrozenTiersInRules or excludeSpecifiedTiersInRules )?

@stash, I think excludedDataTiersForRuleExecution suits better for the purposes. Since Analyzer is boolean switch and this option is string of values - so its name reflects that data tiers in that config would be excluded during rule execution. Initially, users were able to exclude any tier(hit and warm too), but we decided to limit it just to frozen and cold. This name would also help, if decide to include warm tier there too at some point.

When enabled, events from the specified data tiers are not searched during rules executions. This might help to improve rule performance or reduce execution time. If you specify multiple data tiers, separate values with commas. For example: data_frozen,data_cold

I like this option. Only change I would put there is

When configured, events from the specified data tiers are not searched during rules executions. This might help to improve rule performance or reduce execution time. If you specify multiple data tiers, separate values with commas. For example: data_frozen,data_cold

replacing enabled with configured, since it is not boolean switch that can be enabeld/disabled, but config stringm that can have multiple values

@nastasha-solomon

here is a screenshot with changes

rylnd

Code changes look great (just one nit about adding a helper). I smoke tested as well, and things are filtered as expected! Nice work here.

I do see that you have release_note:feature but is this something we're going to be adding dedicated docs for? The tooltip/UI guidance looks good, but it feels like we might need more than that?

rylnd · 2024-07-12T21:54:20Z

x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/eql/eql.test.ts

+      const searchArgs =
+        alertServices.scopedClusterClient.asCurrentUser.eql.search.mock.calls[0][0];
+
+      expect(searchArgs).toHaveProperty(


Ooh, this is a nice matcher I wasn't aware of!

rylnd · 2024-07-12T21:56:31Z

x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/get_filter.ts

+    ? await getDataTierFilter({ uiSettingsClient: services.uiSettingsClient })
+    : [];
+
+  const mergedFilters = [...(filters ? filters : []), ...dataTiersFilters];


We're doing this transformation in a few places. Might be worth adding a utility function like 'mergeArrays`.

all other places it's just simple merge using spread operator

[...array1, ...array2]

No additional logic involved, so I think introducing a separate function that does not do anything apart from spread merge is redundant.
And in fact, would make reading code harder. Since, you'll need to go that function to see what exactly it doing

vitaliidm · 2024-07-15T09:56:26Z

I do see that you have release_note:feature but is this something we're going to be adding dedicated docs for? The tooltip/UI guidance looks good, but it feels like we might need more than that?

@ryland, we have a doc request for that elastic/security-docs#5483

elasticmachine · 2024-07-19T10:00:15Z

⏳ Build in-progress

Buildkite Build
Commit: a80eb93
Cloud Deployment
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186908-a80eb9369fe5
Security Deployment

Failed CI Steps

History

💛 Build #221391 was flaky 5bee3a1
💚 Build #220807 succeeded 407cde2
💛 Build #220654 was flaky e989ad3
💔 Build #220556 failed 1fdfeda

cc @vitaliidm

nastasha-solomon

Just left the one suggestion!

kibana-ci · 2024-07-24T15:24:13Z

💚 Build Succeeded

Buildkite Build
Commit: 9aacf52
Cloud Deployment
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186908-9aacf52919af
Security Deployment

Metrics [docs]

✅ unchanged

History

💔 Build #223455 failed d0eef94
💛 Build #217987 was flaky 5243368
💔 Build #217950 failed eba316d
💔 Build #217860 failed 968405c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm

@timestamp

…om adv settings (#196390) ## Summary - fixes absent data tier filter for ES|QL rule - followup to #186908 ### Demo https://github.com/user-attachments/assets/a6f1290f-ea77-43bf-8def-42712ca5d1b0 ### How to test Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "1m" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "cold": { "min_age": "1m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true }, "set_priority": { "priority": 0 } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details>

@timestamp

…om adv settings (elastic#196390) ## Summary - fixes absent data tier filter for ES|QL rule - followup to elastic#186908 ### Demo https://github.com/user-attachments/assets/a6f1290f-ea77-43bf-8def-42712ca5d1b0 ### How to test Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "1m" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "cold": { "min_age": "1m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true }, "set_priority": { "priority": 0 } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> (cherry picked from commit c79f0ae)

@timestamp

…om adv settings (elastic#196390) ## Summary - fixes absent data tier filter for ES|QL rule - followup to elastic#186908 ### Demo https://github.com/user-attachments/assets/a6f1290f-ea77-43bf-8def-42712ca5d1b0 ### How to test Create a deployment with cold and frozen data tiers and use following commands to create index and ILM <details> <summary>Data tiers commands</summary> ```JSON PUT /_cluster/settings { "persistent": { "indices.lifecycle.poll_interval": "1m" } } PUT /_ilm/policy/filtering_data_tiers { "policy": { "phases": { "frozen": { "min_age": "10m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true } } }, "cold": { "min_age": "1m", "actions": { "searchable_snapshot": { "snapshot_repository": "found-snapshots", "force_merge_index": true }, "set_priority": { "priority": 0 } } }, "hot": { "min_age": "0ms", "actions": { "set_priority": { "priority": 100 } } } } } } PUT /_index_template/filtering_data_tiers_template { "index_patterns": [ "filtering_data_tiers*" ], "template": { "settings": { "index.lifecycle.name": "filtering_data_tiers", "index.lifecycle.rollover_alias": "test-filtering_data_tiers" }, "mappings": { "_meta": { "version": "1.6.0" }, "properties": { "@timestamp": { "type": "date" }, "host": { "properties": { "name": { "type": "keyword", "ignore_above": 1024 } } } } } } } PUT /filtering_data_tiers-000001 { "aliases": { "filtering_data_tiers": { "is_write_index": true } } } POST filtering_data_tiers/_doc { "@timestamp": "2024-07-08T17:00:01.000Z", "host.name": "test-0" } ``` </details> (cherry picked from commit c79f0ae)

[Security Solution][Detection Engine] adds data tier filters to Kiban…

cce930c

…a advanced settings

vitaliidm self-assigned this Jun 25, 2024

vitaliidm added Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. release_note:feature Makes this part of the condensed release notes Team:Detection Engine Security Solution Detection Engine Area 8.16 candidate labels Jun 25, 2024

vitaliidm added 9 commits June 25, 2024 17:43

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

f54d9f0

changes

4d907f1

Merge branch 'de_8_16/add_data_tiers_filtering_adv_settings' of https…

895ff77

…://github.com/vitaliidm/kibana into de_8_16/add_data_tiers_filtering_adv_settings

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

968405c

text

96e2605

Merge branch 'de_8_16/add_data_tiers_filtering_adv_settings' of https…

79894c5

…://github.com/vitaliidm/kibana into de_8_16/add_data_tiers_filtering_adv_settings

text

628623a

text

473f255

fix schemas

7ea7fe1

vitaliidm mentioned this pull request Jun 26, 2024

[Request] Document filtering out cold/frozen data tiers during rule execution elastic/security-docs#5483

Closed

vitaliidm added 14 commits June 26, 2024 15:26

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

eba316d

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

0cf2e8b

fixes

0557fb7

Merge branch 'de_8_16/add_data_tiers_filtering_adv_settings' of https…

5243368

…://github.com/vitaliidm/kibana into de_8_16/add_data_tiers_filtering_adv_settings

fix IM data tiers

6ae1a32

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

ef125ea

simplify types

09223c6

add tests

239be5d

add helpful comment

5cf8c40

adds get_filter tests

2ec0306

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

2c2f361

eql test

c45a136

Merge branch 'de_8_16/add_data_tiers_filtering_adv_settings' of https…

5792f59

…://github.com/vitaliidm/kibana into de_8_16/add_data_tiers_filtering_adv_settings

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

67f38ba

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

21f3e14

vitaliidm added the ci:cloud-deploy Create or update a Cloud deployment label Jul 10, 2024

vitaliidm added 2 commits July 10, 2024 09:31

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

0e3f19c

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

1fdfeda

vitaliidm requested a review from nastasha-solomon July 10, 2024 14:45

vitaliidm commented Jul 10, 2024

View reviewed changes

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

e989ad3

pgayvallet approved these changes Jul 11, 2024

View reviewed changes

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

407cde2

rylnd approved these changes Jul 12, 2024

View reviewed changes

vitaliidm added 2 commits July 15, 2024 11:35

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

5bee3a1

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

a80eb93

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

d0eef94

nastasha-solomon reviewed Jul 24, 2024

View reviewed changes

vitaliidm added 2 commits July 24, 2024 15:33

Merge branch 'main' into de_8_16/add_data_tiers_filtering_adv_settings

85ec3b7

PR feedback

9aacf52

nastasha-solomon approved these changes Jul 24, 2024

View reviewed changes

vitaliidm enabled auto-merge (squash) July 24, 2024 14:58

vitaliidm merged commit 4d5de12 into elastic:main Jul 24, 2024
41 of 42 checks passed

kibanamachine added v8.16.0 backport:skip This commit does not require backporting labels Jul 24, 2024

crespocarlos mentioned this pull request Sep 5, 2024

Do not allow APM UI to query frozen tier #190559

Closed

vitaliidm mentioned this pull request Oct 15, 2024

[Security Solution][Detection Engine] fixes ES|QL data tier filter from adv settings #196390

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] adds data tier filters to Kibana advanced settings #186908

[Security Solution][Detection Engine] adds data tier filters to Kibana advanced settings #186908

vitaliidm commented Jun 25, 2024 •

edited

Loading

vitaliidm Jul 10, 2024

nastasha-solomon Jul 24, 2024 •

edited

Loading

vitaliidm Jul 24, 2024

vitaliidm Jul 24, 2024

rylnd left a comment

rylnd Jul 12, 2024

rylnd Jul 12, 2024

vitaliidm Jul 15, 2024

vitaliidm commented Jul 15, 2024 •

edited

Loading

elasticmachine commented Jul 19, 2024 •

edited

Loading

nastasha-solomon left a comment

kibana-ci commented Jul 24, 2024 •

edited

Loading

[Security Solution][Detection Engine] adds data tier filters to Kibana advanced settings #186908

[Security Solution][Detection Engine] adds data tier filters to Kibana advanced settings #186908

Conversation

vitaliidm commented Jun 25, 2024 • edited Loading

Summary

UI

Demo

How to test

Checklist

vitaliidm Jul 10, 2024

Choose a reason for hiding this comment

nastasha-solomon Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

vitaliidm Jul 24, 2024

Choose a reason for hiding this comment

vitaliidm Jul 24, 2024

Choose a reason for hiding this comment

rylnd left a comment

Choose a reason for hiding this comment

rylnd Jul 12, 2024

Choose a reason for hiding this comment

rylnd Jul 12, 2024

Choose a reason for hiding this comment

vitaliidm Jul 15, 2024

Choose a reason for hiding this comment

vitaliidm commented Jul 15, 2024 • edited Loading

elasticmachine commented Jul 19, 2024 • edited Loading

⏳ Build in-progress

Failed CI Steps

History

nastasha-solomon left a comment

Choose a reason for hiding this comment

kibana-ci commented Jul 24, 2024 • edited Loading

💚 Build Succeeded

Metrics [docs]

History

vitaliidm commented Jun 25, 2024 •

edited

Loading

nastasha-solomon Jul 24, 2024 •

edited

Loading

vitaliidm commented Jul 15, 2024 •

edited

Loading

elasticmachine commented Jul 19, 2024 •

edited

Loading

kibana-ci commented Jul 24, 2024 •

edited

Loading