Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Slow load time for Detection rules page when you have several Indicator Match rules created #82327

Closed
MadameSheema opened this issue Nov 2, 2020 · 3 comments
Closed

[Security Solution] Slow load time for Detection rules page when you have several Indicator Match rules created #82327

MadameSheema opened this issue Nov 2, 2020 · 3 comments
Assignees
@FrankHassanabad
Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Anything related to Security Solution's Detection Rules Feature:Indicator Match Rule Security Solution Indicator Match Rule feature impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.12.0

Comments

@MadameSheema
Copy link
Member

Describe the bug:
Detection rules pages start to be slow when you have several Indicator Match rules created

Kibana/Elasticsearch Stack version:

  • 7.10.0-BC4

** Initial status:**

  • To have at least 6 indicator match rules created with a big look-back time (i.e. 30000 hours) and a small execution time (i.e. 10 seconds)

Steps to reproduce:

  1. Load the Detections rule page

Current behavior:

  • The page takes a long time to be rendered

Expected behavior:

  • The load of the page should not be impacted for the number of rules under execution
@MadameSheema MadameSheema added bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Nov 2, 2020
@peluja1012 peluja1012 added the impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. label Nov 2, 2020
@spong spong added the v7.11.0 label Nov 6, 2020
@peluja1012 peluja1012 added the Feature:Indicator Match Rule Security Solution Indicator Match Rule feature label Nov 17, 2020
@peluja1012 peluja1012 added v7.12.0 and removed v7.11.0 labels Dec 8, 2020
@rylnd rylnd assigned rylnd and FrankHassanabad and unassigned rylnd Jan 7, 2021
@FrankHassanabad FrankHassanabad mentioned this issue Jan 14, 2021
Merged
1 task
FrankHassanabad added a commit that referenced this issue Jan 17, 2021
@FrankHassanabad
[Security Solutions][Detection Engine] Removes duplicate API calls (#…
2abbd80 
…88420)

## Summary

This removes some duplicate API calls to reduce pressure on the backend and speed up querying times within the application for the front end. This fixes some of the issues of #82327, but there are several performance improvements that are going to be needed to help reduce the slowness when you have a system under a lot of pressure.

So far this removes duplication for these API calls when you are on the manage detection rules page:

```ts
api/detection_engine/rules/_find
api/detection_engine/rules/_find_statuses
api/detection_engine/tags
```

<img width="2465" alt="Screen Shot 2021-01-14 at 3 53 21 PM" src="https://user-images.githubusercontent.com/1151048/104662295-c031e080-5687-11eb-92d7-18b9ad355646.png">

* This hides the tags and searches while the page is loading to avoid duplicate calls when the pre-packaged rules counts come back
* This untangles the refetchRules from the refetchPrePackagedRulesStatus as two separate calls to avoid issues we have with re-rendering and re-calling the backend.
 
### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this issue Jan 17, 2021
@FrankHassanabad
[Security Solutions][Detection Engine] Removes duplicate API calls (e…
c4ade57 
…lastic#88420)

## Summary

This removes some duplicate API calls to reduce pressure on the backend and speed up querying times within the application for the front end. This fixes some of the issues of elastic#82327, but there are several performance improvements that are going to be needed to help reduce the slowness when you have a system under a lot of pressure.

So far this removes duplication for these API calls when you are on the manage detection rules page:

```ts
api/detection_engine/rules/_find
api/detection_engine/rules/_find_statuses
api/detection_engine/tags
```

<img width="2465" alt="Screen Shot 2021-01-14 at 3 53 21 PM" src="https://user-images.githubusercontent.com/1151048/104662295-c031e080-5687-11eb-92d7-18b9ad355646.png">

* This hides the tags and searches while the page is loading to avoid duplicate calls when the pre-packaged rules counts come back
* This untangles the refetchRules from the refetchPrePackagedRulesStatus as two separate calls to avoid issues we have with re-rendering and re-calling the backend.
 
### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this issue Jan 17, 2021
@FrankHassanabad @kibanamachine
[Security Solutions][Detection Engine] Removes duplicate API calls (#…
daf5875 
…88420) (#88566)

## Summary

This removes some duplicate API calls to reduce pressure on the backend and speed up querying times within the application for the front end. This fixes some of the issues of #82327, but there are several performance improvements that are going to be needed to help reduce the slowness when you have a system under a lot of pressure.

So far this removes duplication for these API calls when you are on the manage detection rules page:

```ts
api/detection_engine/rules/_find
api/detection_engine/rules/_find_statuses
api/detection_engine/tags
```

<img width="2465" alt="Screen Shot 2021-01-14 at 3 53 21 PM" src="https://user-images.githubusercontent.com/1151048/104662295-c031e080-5687-11eb-92d7-18b9ad355646.png">

* This hides the tags and searches while the page is loading to avoid duplicate calls when the pre-packaged rules counts come back
* This untangles the refetchRules from the refetchPrePackagedRulesStatus as two separate calls to avoid issues we have with re-rendering and re-calling the backend.
 
### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@FrankHassanabad
Copy link
Contributor

This issue is going to persist for a while. The indicator rules are heavy and intensive and running lots of rules at 10 second intervals will be an issue even with optimizations. However, I did take the opportunity to remove duplicate REST based calls which were some of the heaviest so this is an improvement.

Please feel free to either re-open this or open a new one if we need more improvements to performance.

JasonStoltz pushed a commit to JasonStoltz/kibana that referenced this issue Jan 19, 2021
@FrankHassanabad @JasonStoltz
[Security Solutions][Detection Engine] Removes duplicate API calls (e…
67e1065 
…lastic#88420)

## Summary

This removes some duplicate API calls to reduce pressure on the backend and speed up querying times within the application for the front end. This fixes some of the issues of elastic#82327, but there are several performance improvements that are going to be needed to help reduce the slowness when you have a system under a lot of pressure.

So far this removes duplication for these API calls when you are on the manage detection rules page:

```ts
api/detection_engine/rules/_find
api/detection_engine/rules/_find_statuses
api/detection_engine/tags
```

<img width="2465" alt="Screen Shot 2021-01-14 at 3 53 21 PM" src="https://user-images.githubusercontent.com/1151048/104662295-c031e080-5687-11eb-92d7-18b9ad355646.png">

* This hides the tags and searches while the page is loading to avoid duplicate calls when the pre-packaged rules counts come back
* This untangles the refetchRules from the refetchPrePackagedRulesStatus as two separate calls to avoid issues we have with re-rendering and re-calling the backend.
 
### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
banderror pushed a commit to banderror/kibana that referenced this issue Feb 24, 2021
@FrankHassanabad @banderror
[Security Solutions][Detection Engine] Removes duplicate API calls (e…
08ecee9 
…lastic#88420)

## Summary

This removes some duplicate API calls to reduce pressure on the backend and speed up querying times within the application for the front end. This fixes some of the issues of elastic#82327, but there are several performance improvements that are going to be needed to help reduce the slowness when you have a system under a lot of pressure.

So far this removes duplication for these API calls when you are on the manage detection rules page:

```ts
api/detection_engine/rules/_find
api/detection_engine/rules/_find_statuses
api/detection_engine/tags
```

<img width="2465" alt="Screen Shot 2021-01-14 at 3 53 21 PM" src="https://user-images.githubusercontent.com/1151048/104662295-c031e080-5687-11eb-92d7-18b9ad355646.png">

* This hides the tags and searches while the page is loading to avoid duplicate calls when the pre-packaged rules counts come back
* This untangles the refetchRules from the refetchPrePackagedRulesStatus as two separate calls to avoid issues we have with re-rendering and re-calling the backend.
 
### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@ghost
Copy link

ghost commented Mar 4, 2021

Hi @MadameSheema

We have validated this ticket on 7.12.0 BC3 and below are our observations:

Observations:

  • We have created more than 6 indicator rules with look-back time (i.e. 30000 hours) and run every(i.e. 10 seconds)
  • Then navigate to the Detection rule page. Detection rule page loads in 30 seconds.

Screenshots:
Detection_page
ind_rule

Please let us know if we need to re-open this issue

Thanks!!

@ghost
Copy link

ghost commented Mar 30, 2021

Bug Conversion:

Test case already exist for ticket:
https://elastic.testrail.io/index.php?/cases/view/15573

Thanks!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FrankHassanabad FrankHassanabad

Labels
bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Anything related to Security Solution's Detection Rules Feature:Indicator Match Rule Security Solution Indicator Match Rule feature impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.12.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@peluja1012 @rylnd @FrankHassanabad @spong @MadameSheema