[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields #159216

e40pud · 2023-06-07T14:30:12Z

Summary

Original ticket: #158751

These changes improve the rule's exceptions flyout opening experience. We had a few complaints that it is very slow to open it and sometimes it throws an exception about the limited response size.

To fix this, we decided to load extended field's data (conflicts and unmapped info) only when user selects some field instead of fetching this data for all fields on flyout opening.

NOTES:

After these changes we gonna do next steps related to fields loading when user creates/edits rule exceptions:

We will call _fields_for_wildcard WITHOUT include_unmapped=true parameter to fetch all fields specs on exception flyout loading
We will call _fields_for_wildcard WITH include_unmapped=true for only one field when user selects it from the dropdown menu

With these changes we will improve slow exception flyout opening when user has lots of fields which are unmapped in different indices. If for some reason user has a lot of (thousands) conflicting fields around indices then the loading is still might be slow as the _fields_for_wildcard call will return conflicts information even without include_unmapped=true parameter.

…dices with huge amount of fields (elastic#158751)

elasticmachine · 2023-06-07T14:30:18Z

Pinging @elastic/security-solution (Team: SecuritySolution)

e40pud · 2023-06-07T15:53:39Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-09T13:09:12Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-12T15:04:35Z

@elasticmachine merge upstream

…elds

vitaliidm

@e40pud , thanks for working on it. It's going to be a good improvement on performance.

I left 2 comments: one nit(up to you to decide what to do) and one observation

vitaliidm · 2023-06-13T08:31:43Z

...ecurity_solution/public/detection_engine/rule_exceptions/logic/use_exception_flyout_data.tsx

+  }, [memoDataViewId, data.dataViews, setDataViewIndexPatterns, activeSpaceId]);
+
+  // Fetch extended fields information
+  const getExtendedField = useCallback(


nit: getExtendedField can retrieve array of fields, so probably name getExtendedFields would be more suitable

True, will update the name

vitaliidm · 2023-06-13T08:54:36Z

x-pack/plugins/lists/public/exceptions/components/builder/entry_renderer.tsx

+      }
+    };
+    fetchExtendedField();
+  }, [entry.field?.name, getExtendedField]);


I am not very familiar with this functionality, but it looks like there is an initial request that loads all the fields at the flyout's opening.

When I open flyout, I can see the following request

http://localhost:5601/kbn/api/index_patterns/_fields_for_wildcard?pattern=apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Ctraces-apm*%2Cwinlogbeat-*%2C-*elastic-cloud-logs-*%2Cnon-ecs&meta_fields=_source&meta_fields=_id&meta_fields=_index&meta_fields=_score&allow_no_index=true

Which returns 8000 fields

Within the list of the fields, there is a conflicting one, I created

{ "name":"agent.id", "type":"conflict", "esTypes":[ "keyword", "long" ], "searchable":true, "aggregatable":true, "readFromDocValues":false, "conflictDescriptions":{ "keyword":[ ".ds-auditbeat-8.4.2-2023.06.13-000001", ".ds-filebeat-8.5.2-2023.06.13-000001", ".ds-packetbeat-8.5.3-2023.06.13-000001" ], "long":[ "non-ecs" ] }, "metadata_field":false }

When I select, this field in controller, another request is sent, to retrieve this field

http://localhost:5601/kbn/api/index_patterns/_fields_for_wildcard?pattern=apm-*-transaction*%2Cauditbeat-*%2Cendgame-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Ctraces-apm*%2Cwinlogbeat-*%2C-*elastic-cloud-logs-*%2Cnon-ecs&meta_fields=_source&meta_fields=_id&meta_fields=_index&meta_fields=_score&allow_no_index=true&include_unmapped=true&fields=agent.id

The result is identical

{ "name":"agent.id", "type":"conflict", "esTypes":[ "keyword", "long" ], "searchable":true, "aggregatable":true, "readFromDocValues":false, "conflictDescriptions":{ "keyword":[ ".ds-auditbeat-8.4.2-2023.06.13-000001", ".ds-filebeat-8.5.2-2023.06.13-000001", ".ds-packetbeat-8.5.3-2023.06.13-000001" ], "long":[ "non-ecs" ] }, "metadata_field":false }

It looks like, we still retrieve all the fields at the beginning.

Correct, conflicts are always returned if those exist.

Though, the unmapped information will be returned only when include_unmapped=true parameter specified. Users usually have lots of unmapped fields and this can produce huge responses when we call _fields_for_wildcard with include_unmapped=true for all fields.

thanks for explaining and adding to notes.
How comparably larger could be response when include_unmapped=true?
8k items seems quite large already

It can be a really huge difference. Cannot tell exactly in numbers, but what happens when we add include_unmapped info is:

For each field that is unmapped in some indices we would receive conflictDescriptions with the whole list of available indices (some will be in "unmapped" array and the rest in the "field_type" array).

Let's say we have 8k fields and we search through 100 indices. If half of the fields are unmapped in some of the indices for whatever reason (indices store similar but different type of data; we add with time new fields etc.), we will end up with 4k fields * 100 indices extra lines in the response.

e40pud · 2023-06-13T13:59:06Z

@elasticmachine merge upstream

…elds

vitaliidm

Thanks for addressing comments!

dhurley14 · 2023-06-14T15:38:56Z

...ecurity_solution/public/detection_engine/rule_exceptions/logic/use_exception_flyout_data.tsx

+  // Fetch extended fields information
+  const getExtendedFields = useCallback(
+    async (fields: string[]) => {
+      let extendedFields: DataViewField[] = [];


Looks like the response type of getFieldsForIndexPattern is FieldSpec[], not DataViewField[]

kibana/src/plugins/data_views/common/data_views/data_views.ts

Lines 230 to 233 in a1c5603

getFieldsForIndexPattern: (

indexPattern: DataView | DataViewSpec,

options?: GetFieldsOptions | undefined

) => Promise<FieldSpec[]>;

Suggested change

let extendedFields: DataViewField[] = [];

let extendedFields: FieldSpec[] = [];

I will add a separate commit, as I also will need to update imports for these changes

yctercero

Approving to remove security-solution-platform review request. Vitalii approved for team detection-engine.

e40pud · 2023-06-14T16:39:31Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-14T18:58:39Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-14T23:08:16Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-15T06:50:16Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-15T07:55:00Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-15T08:53:59Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-15T09:49:36Z

@elasticmachine merge upstream

…elds

e40pud · 2023-06-15T11:49:56Z

@elasticmachine merge upstream

…elds

kibana-ci · 2023-06-15T12:15:46Z

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

[job] [logs] Explore - Security Solution Tests #2 / Hover actions Adds to timeline
[job] [logs] FTR Configs #13 / Index Management app Index template wizard Mappings step clearing up the Numeric subtype dropdown doesn't break the page

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`lists`	159.1KB	159.5KB	+446.0B
`securitySolution`	10.8MB	10.8MB	+57.0B
total			+503.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	13	15	+2
`securitySolution`	410	414	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	493	497	+4
total			+6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @e40pud

…dices with huge amount of fields (elastic#159216) ## Summary Original ticket: [elastic#158751](elastic#158751) These changes improve the rule's exceptions flyout opening experience. We had a few complaints that it is very slow to open it and sometimes it throws an exception about the limited response size. To fix this, we decided to load extended field's data (conflicts and unmapped info) only when user selects some field instead of fetching this data for all fields on flyout opening. ## NOTES: After these changes we gonna do next steps related to fields loading when user creates/edits rule exceptions: 1. We will call `_fields_for_wildcard` **WITHOUT** `include_unmapped=true` parameter to fetch all fields specs on exception flyout loading 2. We will call `_fields_for_wildcard` **WITH** `include_unmapped=true` for only one field when user selects it from the dropdown menu With these changes we will improve slow exception flyout opening when user has lots of fields which are unmapped in different indices. If for some reason user has a lot of (thousands) conflicting fields around indices then the loading is still might be slow as the `_fields_for_wildcard` call will return conflicts information even without `include_unmapped=true` parameter. --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 31b3477)

kibanamachine · 2023-06-15T13:02:18Z

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…the indices with huge amount of fields (#159216) (#159801) # Backport This will backport the following commits from `main` to `8.8`: - [[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields (#159216)](#159216)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Ievgen Sorokopud <ievgen.sorokopud@elastic.co>

e40pud added 2 commits June 7, 2023 15:35

[Security Solution] Improve rules exception flyout opening for the in…

6736b1e

…dices with huge amount of fields (elastic#158751)

Remove obsolete includeUnmapped use

28c1d32

e40pud added release_note:fix Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. ci:cloud-deploy Create or update a Cloud deployment Team:Detection Engine Security Solution Detection Engine Area labels Jun 7, 2023

e40pud requested review from dhurley14 and yctercero June 7, 2023 14:30

e40pud self-assigned this Jun 7, 2023

e40pud requested review from a team as code owners June 7, 2023 14:30

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

b9fa50a

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

538460a

…elds

yctercero added v8.8.2 v8.9.0 labels Jun 9, 2023

yctercero mentioned this pull request Jun 9, 2023

[Detection Engine] - Exceptions flyout not loading or extremely slow elastic/security-docs#3449

Closed

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

742824b

…elds

vitaliidm reviewed Jun 13, 2023

View reviewed changes

kibanamachine and others added 2 commits June 13, 2023 09:59

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

77cf06d

…elds

Review feedback

af2f91a

e40pud requested a review from vitaliidm June 13, 2023 14:25

vitaliidm approved these changes Jun 13, 2023

View reviewed changes

dhurley14 reviewed Jun 14, 2023

View reviewed changes

yctercero approved these changes Jun 14, 2023

View reviewed changes

Review feedback

e35dff3

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

4083789

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

42da19b

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

c077b2b

…elds

e40pud enabled auto-merge (squash) June 14, 2023 23:08

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

fc6e19d

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

72ca31a

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

ef9cb0f

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

e0a457d

…elds

Merge branch 'main' into security/bugfix/158751-improved-exception-fi…

57f6415

…elds

e40pud merged commit 31b3477 into elastic:main Jun 15, 2023
27 checks passed

kibanamachine mentioned this pull request Jun 15, 2023

[8.8] [Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields (#159216) #159801

Merged

nastasha-solomon mentioned this pull request Jul 5, 2023

UI updated for unmapped fields and fields with type conflicts elastic/security-docs#3527

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields #159216

[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields #159216

e40pud commented Jun 7, 2023 •

edited by kibanamachine

elasticmachine commented Jun 7, 2023

e40pud commented Jun 7, 2023

e40pud commented Jun 9, 2023

e40pud commented Jun 12, 2023

vitaliidm left a comment

vitaliidm Jun 13, 2023

e40pud Jun 13, 2023

vitaliidm Jun 13, 2023

e40pud Jun 13, 2023

vitaliidm Jun 13, 2023

e40pud Jun 13, 2023

e40pud commented Jun 13, 2023

vitaliidm left a comment

dhurley14 Jun 14, 2023

e40pud Jun 14, 2023

yctercero left a comment

e40pud commented Jun 14, 2023

e40pud commented Jun 14, 2023

e40pud commented Jun 14, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

kibana-ci commented Jun 15, 2023 •

edited

ESLint disabled line counts

Total ESLint disabled count

kibanamachine commented Jun 15, 2023

	getFieldsForIndexPattern: (
	indexPattern: DataView \| DataViewSpec,
	options?: GetFieldsOptions \| undefined
	) => Promise<FieldSpec[]>;

	let extendedFields: DataViewField[] = [];
	let extendedFields: FieldSpec[] = [];

[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields #159216

[Security Solution] Improve rules exception flyout opening for the indices with huge amount of fields #159216

Conversation

e40pud commented Jun 7, 2023 • edited by kibanamachine

Summary

NOTES:

elasticmachine commented Jun 7, 2023

e40pud commented Jun 7, 2023

e40pud commented Jun 9, 2023

e40pud commented Jun 12, 2023

vitaliidm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e40pud commented Jun 13, 2023

vitaliidm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yctercero left a comment

Choose a reason for hiding this comment

e40pud commented Jun 14, 2023

e40pud commented Jun 14, 2023

e40pud commented Jun 14, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

e40pud commented Jun 15, 2023

kibana-ci commented Jun 15, 2023 • edited

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

ESLint disabled line counts

Total ESLint disabled count

History

kibanamachine commented Jun 15, 2023

💚 All backports created successfully

Questions ?

e40pud commented Jun 7, 2023 •

edited by kibanamachine

kibana-ci commented Jun 15, 2023 •

edited