[Security Solution][Alerts] Provide more information about rule exception behavior before creation #149149

e40pud · 2023-01-18T18:20:44Z

Summary

These changes surface mapping issues when exceptions are created. We gonna warn the user about type conflicts and unmapped indices.

Tooltip warning inside the field selection dropdown menu:

Warning text underneath the dropdown menu when user picks the field which has mapping issues:

Screen.Recording.2023-01-30.at.12.23.12.mov

Main ticket #146845

…tion behavior before creation (elastic#146845)

elasticmachine · 2023-01-18T18:20:48Z

Pinging @elastic/security-solution (Team: SecuritySolution)

mattkime · 2023-01-18T19:26:45Z

packages/kbn-es-query/src/es_query/types.ts

+  /**
+   * Description of field type conflicts across indices
+   */
+  conflictDescriptions?: Record<string, string[]>;


I'd prefer this remain unchanged. I'm kind of curious how fields are being loaded where this would be helpful.

Reverted these changes. Instead I extended DataViewFieldBase here in the kbn-securitysolution-list-utils package where we use conflict descriptions.

@e40pud It still might be worth having a conversation about how the field list is being loaded to make sure the data views api is doing what it can in this circumstance.

@elastic/security-solution-platform - @e40pud and I had a conversation about how fields are being loaded and it seems like the data views api could be doing more when dealing with unmapped fields. My goal is that you should be able to get the fields you need and related data in a single call. Please reach out to me if you'd like to discuss this further.

Just for a reference, here is the place where I do extra call to get unmapped and conflicts information in data views case.

e40pud · 2023-01-23T13:40:06Z

@elasticmachine merge upstream

e40pud · 2023-01-25T14:26:33Z

@elasticmachine merge upstream

e40pud · 2023-01-26T14:44:29Z

@elasticmachine merge upstream

e40pud · 2023-01-26T18:08:54Z

@elasticmachine merge upstream

e40pud · 2023-01-30T11:41:42Z

@elasticmachine merge upstream

e40pud · 2023-02-06T10:43:18Z

@elasticmachine merge upstream

e40pud · 2023-02-06T10:47:40Z

@nkhristinin We did not discuss such a possibility, but I would expect that if there are any other places that might benefit from this we can introduce these changes there as well. Should be easy to enable it elsewhere in Security Solution. I will talk about it with the team.

logeekal

Overall looks good from investigations perspective. Left a small comment.

logeekal · 2023-02-06T10:58:37Z

x-pack/plugins/timelines/server/search_strategy/index_fields/index.ts

@@ -158,8 +159,12 @@ export const requestIndexFieldSearch = async (
                pattern: index,
              });
            }
+            const fieldCapsOptions = request.includeUnmapped


Overall looks good.

Do you think this warrants adding tests in x-pack/plugins/timelines/server/search_strategy/index_fields/index.test.ts

Good point, I will add tests to cover new changes!

kibana-ci · 2023-02-06T12:42:42Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/securitysolution-list-utils`	150	155	+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lists`	149.3KB	152.1KB	+2.7KB
`securitySolution`	12.9MB	12.9MB	+7.7KB
total			+10.4KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/securitysolution-list-utils`	194	202	+8

History

💔 Build #105566 failed 37d494d
💚 Build #104147 succeeded 35e2e6c
💔 Build #103721 failed 355c513
💛 Build #103112 was flaky 5b59cf3
💔 Build #102999 failed 8ca997c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @e40pud

…tion behavior before creation (elastic#149149) ## Summary These changes surface mapping issues when exceptions are created. We gonna warn the user about type conflicts and unmapped indices. Tooltip warning inside the field selection dropdown menu: <img width="2020" alt="Screenshot 2023-01-18 at 19 01 44" src="https://user-images.githubusercontent.com/2700761/213261684-61d21068-12bc-408f-8d20-1a196e0719a7.png"> Warning text underneath the dropdown menu when user picks the field which has mapping issues: https://user-images.githubusercontent.com/2700761/215467838-5d39ff75-3a2e-44ef-ba89-57cd3975310c.mov Main ticket elastic#146845 --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Main ticket elastic#146845 Addition to PR: elastic#149149

@nastasha-solomon

…152726) ## Summary These changes update warning message that we show to user to indicate index mapping conflicts while selecting a field to build a Rule Exception. New tooltip message: <img width="829" alt="Screenshot 2023-03-06 at 16 18 51" src="https://user-images.githubusercontent.com/2700761/223154197-ee4ed680-5cc1-4b48-82d8-e225aa24519b.png"> [Main ticket](#146845) Addition to [this PR](#149149) cc @nastasha-solomon

@nastasha-solomon

…lastic#152726) ## Summary These changes update warning message that we show to user to indicate index mapping conflicts while selecting a field to build a Rule Exception. New tooltip message: <img width="829" alt="Screenshot 2023-03-06 at 16 18 51" src="https://user-images.githubusercontent.com/2700761/223154197-ee4ed680-5cc1-4b48-82d8-e225aa24519b.png"> [Main ticket](elastic#146845) Addition to [this PR](elastic#149149) cc @nastasha-solomon (cherry picked from commit ce96318)

@nastasha-solomon

…ssage (#152726) (#152755) # Backport This will backport the following commits from `main` to `8.7`: - [[Security Solution][Alerts] Update mapping conflicts warning message (#152726)](#152726)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>

…cts functionality (#151366) ## Summary This PR adds cypress tests to cover new rule exceptions functionality introduced in [this PR](#149149). --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@nastasha-solomon

…lastic#152726) ## Summary These changes update warning message that we show to user to indicate index mapping conflicts while selecting a field to build a Rule Exception. New tooltip message: <img width="829" alt="Screenshot 2023-03-06 at 16 18 51" src="https://user-images.githubusercontent.com/2700761/223154197-ee4ed680-5cc1-4b48-82d8-e225aa24519b.png"> [Main ticket](elastic#146845) Addition to [this PR](elastic#149149) cc @nastasha-solomon

…cts functionality (elastic#151366) ## Summary This PR adds cypress tests to cover new rule exceptions functionality introduced in [this PR](elastic#149149). --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@nastasha-solomon

…lastic#152726) ## Summary These changes update warning message that we show to user to indicate index mapping conflicts while selecting a field to build a Rule Exception. New tooltip message: <img width="829" alt="Screenshot 2023-03-06 at 16 18 51" src="https://user-images.githubusercontent.com/2700761/223154197-ee4ed680-5cc1-4b48-82d8-e225aa24519b.png"> [Main ticket](elastic#146845) Addition to [this PR](elastic#149149) cc @nastasha-solomon

…cts functionality (elastic#151366) ## Summary This PR adds cypress tests to cover new rule exceptions functionality introduced in [this PR](elastic#149149). --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@nastasha-solomon

…152726) ## Summary These changes update warning message that we show to user to indicate index mapping conflicts while selecting a field to build a Rule Exception. New tooltip message: <img width="829" alt="Screenshot 2023-03-06 at 16 18 51" src="https://user-images.githubusercontent.com/2700761/223154197-ee4ed680-5cc1-4b48-82d8-e225aa24519b.png"> [Main ticket](#146845) Addition to [this PR](#149149) cc @nastasha-solomon

…cts functionality (#151366) ## Summary This PR adds cypress tests to cover new rule exceptions functionality introduced in [this PR](#149149). --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@dhurley14

…in Exception Flyout getting stuck in loading state (#158371) ## Summary Original ticket #158110 These changes fixes the issue with the exception flyout which can be stuck in loading state in case `getFieldsForIndexPattern` throws an exception. Fixed by putting the `getFieldsForIndexPattern` call in try/catch. We use this call to fetch extended information about the fields [to show warning to the user in case there are some index issues](#149149). If `getFieldsForIndexPattern` fails and throws an exception we will continue using fields without conflicts/unmapped information. I also, noticed that we do not fetch extended information for the case where user uses index patterns instead of data views. Fixed this issue in `useFetchIndex`. cc @dhurley14 We will need to adjust either of our PRs depending whose changes will go in first :-) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@dhurley14

…in Exception Flyout getting stuck in loading state (elastic#158371) ## Summary Original ticket elastic#158110 These changes fixes the issue with the exception flyout which can be stuck in loading state in case `getFieldsForIndexPattern` throws an exception. Fixed by putting the `getFieldsForIndexPattern` call in try/catch. We use this call to fetch extended information about the fields [to show warning to the user in case there are some index issues](elastic#149149). If `getFieldsForIndexPattern` fails and throws an exception we will continue using fields without conflicts/unmapped information. I also, noticed that we do not fetch extended information for the case where user uses index patterns instead of data views. Fixed this issue in `useFetchIndex`. cc @dhurley14 We will need to adjust either of our PRs depending whose changes will go in first :-) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 1a3cad1)

@dhurley14

…esult in Exception Flyout getting stuck in loading state (#158371) (#158681) # Backport This will backport the following commits from `main` to `8.8`: - [[Security Solution] Failed getFieldsForIndexPattern calls can result in Exception Flyout getting stuck in loading state (#158371)](#158371)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>

[Security Solution][Alerts] Provide more information about rule excep…

d28715e

…tion behavior before creation (elastic#146845)

e40pud requested a review from a team as a code owner January 18, 2023 18:20

e40pud self-assigned this Jan 18, 2023

e40pud requested review from a team as code owners January 18, 2023 18:20

Better namings

d9ea554

mattkime requested changes Jan 18, 2023

View reviewed changes

Review feedback

811b62f

e40pud requested a review from a team as a code owner January 19, 2023 10:49

Fix CI

95cc412

e40pud requested a review from mattkime January 19, 2023 12:35

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

7983d87

mattkime approved these changes Jan 23, 2023

View reviewed changes

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

c49e3e7

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

8ca997c

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

5b59cf3

kibanamachine and others added 2 commits January 30, 2023 06:41

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

41e5b8c

Design feedback

355c513

Merge branch 'main' into security/rule-exceptions/146845-mapping-issues

37d494d

logeekal approved these changes Feb 6, 2023

View reviewed changes

Review feedback

bead0e5

e40pud merged commit 84efdaa into elastic:main Feb 6, 2023

kibanamachine added v8.7.0 backport:skip This commit does not require backporting labels Feb 6, 2023

This was referenced Feb 6, 2023

[Security Solution][Alerts] Rule exception preview #150351

Open

[Security Solution][Alerts] Provide more information about rule exception behavior before creation #146845

Closed

e40pud mentioned this pull request Feb 15, 2023

[Security Solution] Exceptions: Cypress tests to cover mapping conflicts functionality #151366

Merged

e40pud added a commit to e40pud/kibana that referenced this pull request Mar 6, 2023

[Security Solution][Alerts] Update mapping conflicts warning message

6ac8e1c

Main ticket elastic#146845 Addition to PR: elastic#149149

e40pud mentioned this pull request Mar 6, 2023

[Security Solution][Alerts] Update mapping conflicts warning message #152726

Merged

e40pud mentioned this pull request May 24, 2023

[Security Solution] Failed getFieldsForIndexPattern calls can result in Exception Flyout getting stuck in loading state #158371

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Alerts] Provide more information about rule exception behavior before creation #149149

[Security Solution][Alerts] Provide more information about rule exception behavior before creation #149149

e40pud commented Jan 18, 2023 •

edited

elasticmachine commented Jan 18, 2023

mattkime Jan 18, 2023

e40pud Jan 19, 2023

mattkime Jan 19, 2023

mattkime Jan 19, 2023

e40pud Jan 19, 2023

e40pud commented Jan 23, 2023

e40pud commented Jan 25, 2023

e40pud commented Jan 26, 2023

e40pud commented Jan 26, 2023

e40pud commented Jan 30, 2023

e40pud commented Feb 6, 2023

e40pud commented Feb 6, 2023

logeekal left a comment

logeekal Feb 6, 2023

e40pud Feb 6, 2023

kibana-ci commented Feb 6, 2023 •

edited

API count

[Security Solution][Alerts] Provide more information about rule exception behavior before creation #149149

[Security Solution][Alerts] Provide more information about rule exception behavior before creation #149149

Conversation

e40pud commented Jan 18, 2023 • edited

Summary

elasticmachine commented Jan 18, 2023

mattkime Jan 18, 2023

Choose a reason for hiding this comment

e40pud Jan 19, 2023

Choose a reason for hiding this comment

mattkime Jan 19, 2023

Choose a reason for hiding this comment

mattkime Jan 19, 2023

Choose a reason for hiding this comment

e40pud Jan 19, 2023

Choose a reason for hiding this comment

e40pud commented Jan 23, 2023

e40pud commented Jan 25, 2023

e40pud commented Jan 26, 2023

e40pud commented Jan 26, 2023

e40pud commented Jan 30, 2023

e40pud commented Feb 6, 2023

e40pud commented Feb 6, 2023

logeekal left a comment

Choose a reason for hiding this comment

logeekal Feb 6, 2023

Choose a reason for hiding this comment

e40pud Feb 6, 2023

Choose a reason for hiding this comment

kibana-ci commented Feb 6, 2023 • edited

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

API count

History

e40pud commented Jan 18, 2023 •

edited

kibana-ci commented Feb 6, 2023 •

edited