[Security Solution] Improve find rule and find rule status route performance #99678

marshallmain · 2021-05-10T19:19:43Z

Summary

With the implementation of aggregations in saved objects, we can now fetch statuses for multiple rules in one request. This PR implements a findBulk method on the RuleStatusClient, adds a bulk "get rule actions" method, and parellelizes the requests in the find rule status route. We can further improve performance in the rule status route by replacing the N get calls in getFailingRules with a single getBulk call if bulk AlertsClient methods are added (#99216). With an experimental bulkGet implementation, this cut the rule status API request time to ~300ms (an additional ~60% reduction over the performance in this PR).

Find route

Rows	master	PR branch
20	100ms	100ms
1000	5s	500ms

Find status route

Rows	master	PR branch
20	200ms	120ms
1000	3.5s	800ms

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…quests

elasticmachine · 2021-05-10T19:22:06Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2021-05-10T19:22:06Z

Pinging @elastic/security-solution (Team: SecuritySolution)

marshallmain · 2021-05-10T19:22:23Z

@elasticmachine merge upstream

marshallmain · 2021-05-10T19:29:25Z

src/core/server/saved_objects/service/lib/aggregations/aggs_types/metrics_aggs.ts

@@ -68,7 +68,7 @@ export const metricsAggsSchemas: Record<string, ObjectType> = {
    stored_fields: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
    from: s.maybe(s.number()),
    size: s.maybe(s.number()),
-    sort: s.maybe(s.oneOf([s.literal('asc'), s.literal('desc')])),
+    sort: s.maybe(s.any()),


The schema here was too strict - string values in sort are used as field names, so this only allowed sorting on fields named asc or desc. The JS Elasticsearch client types allow the full range of Sort options for top hits aggs (https://github.com/elastic/elasticsearch-js/blob/master/api/types.d.ts#L11982)

Sorry, I would really like to avoid any in the aggregation schemas. I'm very fine changing the sort schema to get closer to the real type (it do not need to be exhaustive here, only to match your needs), but the whole goal of the aggs validation is to not allow arbitrary values.

export type Sort = SortCombinations | Array<SortCombinations> export type SortCombinations = Field | SortContainer | SortOrder export type SortOrder = 'asc' | 'desc' | '_doc'; export interface Field { name: string; type: string; esTypes?: string[]; aggregatable: boolean; filterable: boolean; searchable: boolean; subType?: FieldSubType; } export interface SortContainerKeys { _score?: ScoreSort _doc?: ScoreSort _geo_distance?: GeoDistanceSort _script?: ScriptSort } export type SortContainer = SortContainerKeys | { [property: string]: FieldSort | SortOrder }

From your code, it seems you're using the SortContainer format

I would introduce a SortSchema, and also use it in other aggregations where we have a sort options, e.g terms.

// note: these schemas are not exhaustive. See the `Sort` type of `@elastic/elasticsearch` if you need to enhance it. const sortOrderSchema = s.oneOf([s.literal('asc'), s.literal('desc'), s.literal('_doc')]); const sortContainerSchema = s.recordOf(s.string(), sortOrderSchema); const sortCombinationsSchema = s.oneOf([sortContainerSchema, sortOrderSchema]) const sortSchema = s.oneOf([sortCombinationsSchema, s.arrayOf(sortCombinationsSchema)])

You can create a src/core/server/saved_objects/service/lib/aggregations/aggs_types/common_schemas.ts file for that.

No problem, thanks for the feedback! I added the file you suggested with some of the reusable sort schemas and used them to enhance the terms.order and top hits sort validation.

marshallmain · 2021-05-11T04:54:00Z

@elasticmachine merge upstream

pgayvallet · 2021-05-11T07:32:23Z

src/core/server/saved_objects/service/lib/aggregations/aggs_types/metrics_aggs.ts

@@ -68,7 +68,7 @@ export const metricsAggsSchemas: Record<string, ObjectType> = {
    stored_fields: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
    from: s.maybe(s.number()),
    size: s.maybe(s.number()),
-    sort: s.maybe(s.oneOf([s.literal('asc'), s.literal('desc')])),
+    sort: s.maybe(s.any()),


Sorry, I would really like to avoid any in the aggregation schemas. I'm very fine changing the sort schema to get closer to the real type (it do not need to be exhaustive here, only to match your needs), but the whole goal of the aggs validation is to not allow arbitrary values.

export type Sort = SortCombinations | Array<SortCombinations> export type SortCombinations = Field | SortContainer | SortOrder export type SortOrder = 'asc' | 'desc' | '_doc'; export interface Field { name: string; type: string; esTypes?: string[]; aggregatable: boolean; filterable: boolean; searchable: boolean; subType?: FieldSubType; } export interface SortContainerKeys { _score?: ScoreSort _doc?: ScoreSort _geo_distance?: GeoDistanceSort _script?: ScriptSort } export type SortContainer = SortContainerKeys | { [property: string]: FieldSort | SortOrder }

From your code, it seems you're using the SortContainer format

I would introduce a SortSchema, and also use it in other aggregations where we have a sort options, e.g terms.

// note: these schemas are not exhaustive. See the `Sort` type of `@elastic/elasticsearch` if you need to enhance it. const sortOrderSchema = s.oneOf([s.literal('asc'), s.literal('desc'), s.literal('_doc')]); const sortContainerSchema = s.recordOf(s.string(), sortOrderSchema); const sortCombinationsSchema = s.oneOf([sortContainerSchema, sortOrderSchema]) const sortSchema = s.oneOf([sortCombinationsSchema, s.arrayOf(sortCombinationsSchema)])

You can create a src/core/server/saved_objects/service/lib/aggregations/aggs_types/common_schemas.ts file for that.

…ibana into rules-status-aggs-no-bulk

marshallmain · 2021-05-13T21:41:29Z

@elasticmachine merge upstream

FrankHassanabad · 2021-05-14T14:21:22Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts

@@ -10,7 +10,7 @@ import moment from 'moment';
 import uuidv5 from 'uuid/v5';
 import dateMath from '@elastic/datemath';
 import type { estypes } from '@elastic/elasticsearch';
-import { isEmpty, partition } from 'lodash';
+import { chunk, isEmpty, partition } from 'lodash';


It's ok if you want to use lodash, although still most people pull from lodash/fp but I don't see a big issue unless I see someone using one the mutatious things from lodash.

FrankHassanabad · 2021-05-14T14:24:30Z

...ns/security_solution/server/lib/detection_engine/signals/rule_status_saved_objects_client.ts

+    const buckets = get(results, 'aggregations.alertIds.buckets');
+    return buckets.reduce((acc: Record<string, unknown>, bucket: unknown) => {
+      const key = get(bucket, 'key');
+      const hits = get(bucket, 'most_recent_statuses.hits.hits');


nit: I would try to cast here if possible maybe? const hits: someType even if that type was something loose such as Record<string, unknown>. I didn't play with it, but less any is good where/when we can use unknown or some other type.

FrankHassanabad

Looks straight forward enough for security solutions. Thanks for the improvements, good luck with the core pieces and other feedback on it.

pgayvallet

LGTM, thanks for the changes

peluja1012 · 2021-05-20T16:23:47Z

@elasticmachine merge upstream

kibanamachine · 2021-05-20T16:23:48Z

merge conflict between base and head

kibanamachine · 2021-05-28T15:48:15Z

💚 Build Succeeded

Metrics [docs]

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`canvas`	29	25	-4
`crossClusterReplication`	8	6	-2
`fleet`	22	20	-2
`globalSearch`	4	2	-2
`indexManagement`	12	7	-5
`infra`	261	149	-112
`lens`	67	45	-22
`licensing`	18	15	-3
`lists`	239	236	-3
`maps`	286	208	-78
`ml`	121	115	-6
`monitoring`	109	56	-53
`securitySolution`	386	342	-44
`stackAlerts`	101	95	-6
total			-342

History

💚 Build #125909 succeeded e8c4511
💚 Build #125466 succeeded d7997e1
💔 Build #125457 failed 17e72a0
💔 Build #125237 failed b443b9f
💔 Build #125160 failed 5de081a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…ormance (elastic#99678) * Fetch rule statuses using single aggregation instead of N separate requests * Optimize _find API and _find_statuses * Merge alerting framework errors into rule statuses * Add sortSchema for top hits agg, update terms.order schema Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-05-28T15:52:17Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…ormance (#99678) (#100912) * Fetch rule statuses using single aggregation instead of N separate requests * Optimize _find API and _find_statuses * Merge alerting framework errors into rule statuses * Add sortSchema for top hits agg, update terms.order schema Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

* master: (77 commits) [RAC][Security Solution] Register Security Detection Rules with Rule Registry (elastic#96015) [Enterprise Search] Log warning for Kibana/EntSearch version mismatches (elastic#100809) updating the saved objects test to include more saved object types (elastic#100828) [ML] Fix categorization job view examples link when datafeed uses multiple indices (elastic#100789) Fixing ES archive mapping failure (elastic#100835) Fix bug with Observability > APM header navigation (elastic#100845) [Security Solution][Endpoint] Add event filters summary card to the fleet endpoint tab (elastic#100668) [Actions] Taking space id into account when creating email footer link (elastic#100734) Ensure comments on parameters in arrow functions are captured in the docs and ci metrics. (elastic#100823) [Security Solution] Improve find rule and find rule status route performance (elastic#99678) [DOCS] Adds video to introduction (elastic#100906) [Fleet] Improve combo box for fleet settings (elastic#100603) [Security Solution][Endpoint] Endpoint generator and data loader support for Host Isolation (elastic#100813) [DOCS] Adds Lens video (elastic#100898) [TSVB] [Table tab] Fix "Math" aggregation (elastic#100765) chore(NA): moving @kbn/io-ts-utils into bazel (elastic#100810) [Alerting] Adding feature flag for enabling/disabling rule import and export (elastic#100718) [TSVB] Fix Upgrading from 7.12.1 to 7.13.0 breaks TSVB (elastic#100864) [Lens] Adds dynamic table cell coloring (elastic#95217) [Security Solution][Endpoint] Do not display searchbar in security-trusted apps if there are no items (elastic#100853) ...

marshallmain added 7 commits April 28, 2021 14:51

Fetch rule statuses using single aggregation instead of N separate re…

361be5e

…quests

Optimize _find API and _find_statuses more

6ebe345

Merge alerting framework errors into rule statuses

2ef8df5

Extract shared function, remove console logs

f76456f

Remove AlertsClient getBulk experimental implementation

391eec4

Cleanup types and tests

3773478

Remove commented code

543f230

marshallmain requested review from a team as code owners May 10, 2021 19:19

marshallmain added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detections and Resp Security Detection Response Team v7.14.0 v8.0.0 labels May 10, 2021

marshallmain requested a review from a team May 10, 2021 19:22

Merge branch 'master' into rules-status-aggs-no-bulk

5de081a

marshallmain commented May 10, 2021

View reviewed changes

Fix unit tests

8700301

Merge branch 'master' into rules-status-aggs-no-bulk

b443b9f

pgayvallet requested changes May 11, 2021

View reviewed changes

marshallmain added 3 commits May 11, 2021 17:11

Add sortSchema for top hits agg, update terms.order schema

292a360

Merge branch 'rules-status-aggs-no-bulk' of github.com:marshallmain/k…

17e72a0

…ibana into rules-status-aggs-no-bulk

Remove unused import

d7997e1

marshallmain requested a review from pgayvallet May 13, 2021 21:41

Merge branch 'master' into rules-status-aggs-no-bulk

e8c4511

FrankHassanabad reviewed May 14, 2021

View reviewed changes

FrankHassanabad approved these changes May 14, 2021

View reviewed changes

pgayvallet approved these changes May 17, 2021

View reviewed changes

spong requested a review from a team May 25, 2021 16:07

Merge branch 'master' into rules-status-aggs-no-bulk

2316fae

marshallmain added the auto-backport Deprecated - use backport:version if exact versions are needed label May 28, 2021

marshallmain merged commit 7f6d7b3 into elastic:master May 28, 2021

kibanamachine mentioned this pull request May 28, 2021

[7.x] [Security Solution] Improve find rule and find rule status route performance (#99678) #100912

Merged

xcrzx mentioned this pull request Jun 15, 2021

[Security Solution] Detection Rule Management is very slow #98287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Improve find rule and find rule status route performance #99678

[Security Solution] Improve find rule and find rule status route performance #99678

marshallmain commented May 10, 2021 •

edited

Loading

elasticmachine commented May 10, 2021

elasticmachine commented May 10, 2021

marshallmain commented May 10, 2021

marshallmain May 10, 2021 •

edited

Loading

pgayvallet May 11, 2021 •

edited

Loading

marshallmain May 12, 2021

marshallmain commented May 11, 2021

pgayvallet May 11, 2021 •

edited

Loading

marshallmain commented May 13, 2021

FrankHassanabad May 14, 2021

FrankHassanabad May 14, 2021

FrankHassanabad left a comment

pgayvallet left a comment

peluja1012 commented May 20, 2021

kibanamachine commented May 20, 2021

kibanamachine commented May 28, 2021

References to deprecated APIs

kibanamachine commented May 28, 2021

[Security Solution] Improve find rule and find rule status route performance #99678

[Security Solution] Improve find rule and find rule status route performance #99678

Conversation

marshallmain commented May 10, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented May 10, 2021

elasticmachine commented May 10, 2021

marshallmain commented May 10, 2021

marshallmain May 10, 2021 • edited Loading

Choose a reason for hiding this comment

pgayvallet May 11, 2021 • edited Loading

Choose a reason for hiding this comment

marshallmain May 12, 2021

Choose a reason for hiding this comment

marshallmain commented May 11, 2021

pgayvallet May 11, 2021 • edited Loading

Choose a reason for hiding this comment

marshallmain commented May 13, 2021

FrankHassanabad May 14, 2021

Choose a reason for hiding this comment

FrankHassanabad May 14, 2021

Choose a reason for hiding this comment

FrankHassanabad left a comment

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

peluja1012 commented May 20, 2021

kibanamachine commented May 20, 2021

kibanamachine commented May 28, 2021

💚 Build Succeeded

Metrics [docs]

References to deprecated APIs

History

kibanamachine commented May 28, 2021

💚 Backport successful

marshallmain commented May 10, 2021 •

edited

Loading

marshallmain May 10, 2021 •

edited

Loading

pgayvallet May 11, 2021 •

edited

Loading

pgayvallet May 11, 2021 •

edited

Loading