Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Setting a large default rule interval #171814

Draft
wants to merge 65 commits into
base: main
Choose a base branch
from

Conversation

VidhiRambhia
Copy link
Contributor

@VidhiRambhia VidhiRambhia commented Nov 23, 2023

Resolves: #153689

Summary

The issue: In many tests we use a short rule execution interval ('5m' or less). This leads to rules getting scheduled for execution many times during a test run. It unnecessarily hogs on CI resources. For most tests we just want a single execution.

This PR:

  • Sets 24h as the default interval for all rule creation utils used in Cypress tests and in API integration tests
  • Overrides the default 24h interval for a few tests that need a shorter interval.

@VidhiRambhia VidhiRambhia marked this pull request as ready for review November 29, 2023 20:16
@VidhiRambhia VidhiRambhia requested a review from a team as a code owner November 29, 2023 20:16
@VidhiRambhia
Copy link
Contributor Author

Hi @banderror,

Please let me know your thoughts on this. Thanks!

Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @VidhiRambhia, I reviewed the change and left a comment.

x-pack/test/security_solution_cypress/config.ts Outdated Show resolved Hide resolved
@banderror banderror changed the title Security Solution | Setting a large default rule interval [Security Solution] Setting a large default rule interval Nov 30, 2023
@banderror banderror added test test_ui_functional release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team labels Nov 30, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@VidhiRambhia VidhiRambhia requested review from a team as code owners December 1, 2023 19:02
@nikitaindik
Copy link
Contributor

/ci

Copy link
Contributor

@PhilippeOberti PhilippeOberti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the Threat Hunting team

Copy link
Contributor

@dhurley14 dhurley14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with the caveat I think with these changes it would be good to see a test case in x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_creation/trial_license_complete_tier/create_rules.ts that removes the interval, creates a rule with no interval defined, and checks that it still defaults to 5m.

@@ -1624,7 +1624,7 @@ export default ({ getService }: FtrProviderContext): void => {
{
type: BulkActionEditTypeEnum.set_rule_actions,
value: {
throttle: '1h',
throttle: '1d',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a preference here for mixing '1d' values with '24h' values? Should we standardize on one vs the other?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used 24h everywhere, but here it had to be 1d, because throttle has to be on of these

const ThrottleForBulkActions = z.enum(['rule', '1h', '1d', '7d']);

@nikitaindik
Copy link
Contributor

Approving with the caveat I think with these changes it would be good to see a test case in x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_creation/trial_license_complete_tier/create_rules.ts that removes the interval, creates a rule with no interval defined, and checks that it still defaults to 5m.

Thanks @dhurley14! Added in this commit.

@nikitaindik nikitaindik requested a review from dplumlee June 3, 2024 13:59
Copy link
Member

@MadameSheema MadameSheema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-engineering-productivity changes LGTM

Copy link
Contributor

@dplumlee dplumlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates @nikitaindik! Rule management changes lgtm, just left a question

@@ -145,13 +145,17 @@ export default ({ getService }: FtrProviderContext): void => {
const rule: ReturnType<typeof getSimpleRule> = {
...getSimpleRule('rule-1'),
throttle: '1h', // <-- throttle makes this a scheduled action
interval: '50m', // has to be less than "throttle"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this have to be less than throttle here but equal in other places? Just because we're testing action logic?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, you're right. I have re-checked and it looks like we can use 24h in both places.

@banderror
Copy link
Contributor

@elasticmachine merge upstream

@banderror
Copy link
Contributor

@elasticmachine run elasticsearch-ci/docs

@banderror
Copy link
Contributor

/ci

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this file we have mocks with interval == 5m. Is this on purpose?

  • getUpdateNewTermsSchemaMock - this function seems to be unused. Can we update the interval here to 24h?
  • getCreateThreatMatchRulesSchemaMock - this one is used, but only from unit tests. Can we update the interval here to 24h?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Changed to 24h in both functions in this commit

Comment on lines 151 to 152
interval: '24h',
from: 'now-6m',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interval and from fields are tightly coupled. In the app, the from field will always be now - (interval + lookback). Here, the implied lookback was 1 minute before the change to 24h.

I'd suggest to update the from field according to the new interval value in all the places where we're updating the interval in this PR.

Suggested change
interval: '24h',
from: 'now-6m',
interval: '24h', // or 1440 mins
from: 'now-1450m', // lookback of 10 mins in addition to the interval of 1440 mins

We can keep the tests and mocks where from is a constant from the very past, where likely it was done on purpose to be able to read source events with static timestamps (e.g. from an es archive):

  interval: '24h',
  from: '1900-01-01T00:00:00.000Z',

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Updated in this commit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a general comment not related to this file. I can still see many instances of the 5-minute interval in test mocks, integration tests, and e2e tests.

Common and server-side mocks:

There are many interval: '5m' instances in the security_solution_api_integration/test_suites/detections_response folder.

There's also one in security_solution_cypress/cypress/e2e/detection_response:

Finally, there's a lot of them in unit tests.

While these intervals might not be affecting those tests where we create enabled rules and run them, keeping hundreds of interval: '5m' instances in the code seems off, unnecessary and potentially unsafe.

I'd be fine with merging this PR without further large-scale refactoring, if we create a new tech debt ticket for addressing this in the future. As part of this ticket, we might want to consolidate all our rule mocks under security_solution/common/api/detection_engine/model/rule_schema.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another general comment: did we run the changed tests in the flaky test runner?

@mistic
Copy link
Member

mistic commented Jun 4, 2024

buildkite test this

@nikitaindik
Copy link
Contributor

Thanks for the feedback @banderror! I'll update "from" values as you suggested, add a new Flaky Test Runner run and will create a ticket for further refactoring. Changing the PR status back to Draft for now.

@nikitaindik nikitaindik marked this pull request as draft June 5, 2024 08:26
@@ -11,5 +11,6 @@ export default function ({ loadTestFile }: FtrProviderContext) {
loadTestFile(require.resolve('./create_rules'));
loadTestFile(require.resolve('./create_new_terms'));
loadTestFile(require.resolve('./preview_rules'));
loadTestFile(require.resolve('./create_rules_bulk'));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test file was not imported anywhere for some reason. Even though the tests in this file seem legit and not a duplicate of some other tests. So I imported it here.

@nikitaindik
Copy link
Contributor

/ci

@kibana-ci
Copy link
Collaborator

kibana-ci commented Jun 5, 2024

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #99 / Monitoring Endpoints Elasticsearch "before all" hook in "Elasticsearch"
  • [job] [logs] FTR Configs #99 / Monitoring Endpoints Kibana Overview Package (metrics-*) should get kibana rules at cluster level
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions with frequencies it does not change actions frequency attributes when 'throttle' is 'undefined'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions with frequencies it does not change actions frequency attributes when 'throttle' is 'undefined'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 300s' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 300s' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 3h' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 3h' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 4d' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 4d' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 5m' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess @serverless @skipInServerlessMKI update_rules update rules per-action frequencies @skipInServerless actions without frequencies it correctly transforms 'throttle = 5m' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '10h'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '10h'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '2d'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '2d'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '321s'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '321s'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '6m'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is '6m'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'no_actions'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'no_actions'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'rule'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'rule'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'undefined'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions with frequencies it does not change actions frequency attributes when 'throttle' is 'undefined'
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 300s' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 300s' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 3h' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 3h' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 4d' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 4d' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 5m' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies actions without frequencies it correctly transforms 'throttle = 5m' and sets it as a frequency of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 1h' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 1h' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 430s' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 430s' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 7m' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 7m' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 8d' and overrides frequency attribute of each action
  • [job] [logs] FTR Configs #53 / Rules Management - Rule Update APIs @ess update_rules_bulk bulk per-action frequencies some actions with frequencies it correctly transforms 'throttle = 8d' and overrides frequency attribute of each action

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nikitaindik @VidhiRambhia

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💝community release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test_ui_functional test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Security Solution] Cypress: set a large default rule interval