Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Detection Engine Test Automation and Coverage #153633

Open
23 of 54 tasks
banderror opened this issue Mar 24, 2023 · 0 comments
Open
23 of 54 tasks

[Security Solution] Detection Engine Test Automation and Coverage #153633

banderror opened this issue Mar 24, 2023 · 0 comments
Assignees
@banderror @yctercero
Labels
epic Feature:Detection Alerts Security Solution Detection Alerts Feature Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration test-coverage issues & PRs for improving code test coverage v8.10.0 v8.11.0 v8.12.0

Comments

@banderror
Copy link
Contributor

banderror commented Mar 24, 2023 •
edited

Related to: https://github.com/elastic/security-team/issues/6482, https://github.com/elastic/security-team/issues/6706, https://github.com/elastic/security-team/issues/6607, https://github.com/elastic/security-team/issues/6754 (internal issues)

Summary

We need to start stabilizing and improving our automated tests and increasing the test coverage for our features. Reasons:

  • Having stable, non-flaky tests with sufficient coverage of our features is very important for going live with Serverless. This is a very high priority.
  • We've been lacking help from QA engineers, for reasons outside of our control, and unfortunately, we can't rely on manual testing -- neither from their side nor from ourselves. We need all tests to be automated.

Goals:

  1. Flakiness. Make Cypress and API integration tests stable: reduce flakiness to zero.
  2. Serverless tests. We should create a dedicated suite of E2E and API integration tests that will be run against Serverless environments. We should reuse our existing tests instead of duplicating them.
  3. Structure. Improve the folder structure of the tests to be able to fix the ownership problem. Split tests into semantically meaningful and cohesive groups and FTR configs. Create two new Cypress configs for Detection Engine and Rule Management teams.
  4. Ownership. Fix issues with the code ownership of Cypress and API integration tests. Update the CODEOWNERS file.
  5. Speed. Make Cypress and API integration tests faster: both locally (speed) and on CI (parallelization). Each parallel task should run under 40 minutes.
  6. CI/CD. Enable running Cypress tests for all PRs on CI.
  7. Knowledge sharing. Write developer docs on best practices for writing Cypress and API integration tests. Start sharing knowledge across AET and whole Security.
  8. Coverage. Increase the test coverage for every feature each of the area teams owns: @elastic/security-detection-engine, @elastic/security-detection-rule-management. For each feature, we should write a test plan and cover its business logic by any/all types of tests: E2E, integration, and unit.
  9. Maintainability. Refactor Cypress and API integration tests to improve their maintainability.

Sub-tasks

Flakiness
Beta Give feedback
  1. [Security Solution] Flaky tests #153667
    15 of 15
    8.10 candidate Meta Team: SecuritySolution Team:Detections and Resp Team:Threat Hunting
  2. [Security Solution] Fix flaky tests: Detection Engine area #161506
    8 of 8
    8.10 candidate Meta Team: SecuritySolution Team:Detection Engine Team:Detections and Resp technical debt test test-api-integration test_ui_functional v8.10.0
    WafaaNasr e40pud
  3. [Security Solution] Fix flaky tests: Rule Management area #161507
    18 of 18
    8.10 candidate Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test-api-integration test_ui_functional v8.10.0
    jpdjere maximpn
  4. [Security Solution] Fix flaky tests: Detection and Response team 8.11-8.12 #165359
    9 of 9
    8.12 candidate Meta Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp technical debt test test-api-integration test_ui_functional v8.11.0 v8.12.0
    jpdjere maximpn
  5. [Security Solution] Triage and skip new flakes in Rule Management tests #173469
    6 of 6
    8.12 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-failure-flaky
    banderror
  6. [Security Solution] Fix flaky tests: Rule Management team 8.13 #173731
    13 of 13
    8.13 candidate Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test-api-integration test_ui_functional
    maximpn
  7. [Security Solution] Fix flaky tests: Rule Management team 8.15 #176909
    1 of 7
    8.15 candidate Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test-api-integration test_ui_functional
  8. [Security Solution] Fix flaky tests: common tasks #161505
    4 of 9
    Meta Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp Team:Threat Hunting technical debt test test-api-integration test_ui_functional
  9. [Security Solution] Collect ideas for reducing flakiness and increasing speed of Cypress tests #153655
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp performance technical debt test test_ui_functional
    banderror
  10. [Security Solution] Create a knowledge base of common flake problems with tests #161516
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp docs test
  11. [Security Solution] Improving test automation reliability #173508
    6 of 11
    Team: SecuritySolution
  12. [Security Solution] Guarantee Cypress Reliability: Stabilizing Using Clean Retries #174247
    Team: SecuritySolution
  13. [Detection Engine] Remove before and after hooks from Cypress tests #175021
    Team: SecuritySolution Team:Detection Engine
  14. [Rule Management] Remove before and after hooks from Cypress tests #175022
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test
  15. [Detection Engine] Audit clean-up of data in Cypress #175097
    Team: SecuritySolution Team:Detection Engine
  16. [Rule Management] Audit clean-up of data in Cypress #175098
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test

Speed
Beta Give feedback
  1. [Security Solution] Split Cypress test files that run more than 15 mins into smaller files #166149
    8.13 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test_ui_functional

Structure and Ownership
Beta Give feedback
  1. [Security Solution] Identify Detection Engine Cypress tests ownership #153658
    Team:Detection Engine
    yctercero
  2. [Security Solution] Identify Rule Management Cypress tests ownership #153656
    8.10 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test_ui_functional
    banderror
  3. [Security Solution] Restructure Cypress tests and fix code ownership #153645
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp refactoring technical debt test test_ui_functional v8.10.0
  4. [Security Solution] Cypress: extract rules-related code from tasks/common into separate files #153695
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp refactoring technical debt test test_ui_functional
  5. [Security Solution] Split Cypress execution on CI between Detections and Response and Threat Hunting #153661
    8.12 candidate Team: SecuritySolution Team:Detections and Resp Team:Threat Hunting technical debt test test_ui_functional
    MadameSheema
  6. [Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902
    8.13 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring technical debt test test-api-integration
    WafaaNasr yctercero
  7. [Security Solution] Restructure API integration tests based on their required license type #151877
    2 of 5
    8.13 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring technical debt test test-api-integration

Serverless tests
Beta Give feedback
  1. [Security Solution] Find a way to reuse tests between ESS and Serverless suites #161537
    8.11 candidate Team: SecuritySolution Team:Defend Workflows Team:Detections and Resp Team:Threat Hunting test test-api-integration test_ui_functional
    WafaaNasr
  2. [Security Solution] Skip all existing Cypress tests in Serverless: Detection & Response area #164441
    8.11 candidate Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp test
    banderror
  3. [Security Solution] Enable Cypress tests in Serverless: Detection Engine area #161539
    8.12 candidate Team: SecuritySolution Team:Detection Engine Team:Detections and Resp test test-coverage
    jpdjere yctercero
  4. [Security Solution] Enable Cypress tests in Serverless: Rule Management area #161540
    8.12 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage
    maximpn
  5. [Security Solution] Enable API integration tests in Serverless: Detection Engine area #164445
    8.12 candidate Team: SecuritySolution Team:Detection Engine Team:Detections and Resp test test-coverage
    WafaaNasr
  6. [Security Solution] Enable API integration tests in Serverless: Rule Management area #164448
    8.13 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage v8.12.0
  7. [Security Solution] Support testing with different roles in Serverless Cypress and API integration tests #164451
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp test
    maximpn
  8. [Security Solution] Enable API integration tests in Serverless, second quality gate #169185
    4 of 24
    Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp consider-next test test-coverage
    yctercero
  9. [Security Solution] Fix tests in MKI: Rule Management team 8.14 #179264
    0 of 15
    8.14 candidate Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test_ui_functional
    banderror

CI/CD
Beta Give feedback
  1. [Security Solution] Stabilize Cypress #153664
    8.11 candidate Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Threat Hunting Theme: consistent_performant_ux performance technical debt test test_ui_functional
    MadameSheema marshallmain
    maximpn michaelolo24 stephmilovic yctercero
  4. [Security Solution] Preparing Cypress for the second quality gate - latest steps #173327
    Team: SecuritySolution
    MadameSheema

Knowledge sharing
Beta Give feedback
  1. [Security Solution] Create a knowledge base of common flake problems with tests #161516
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp docs test
  2. [Security Solution] Document best practices for test automation in general #153682
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp documentation technical debt test
  3. [Security Solution] Document best practices for writing Cypress tests #153662
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp documentation technical debt test test_ui_functional
  4. [Security Solution] Document best practices for writing API integration tests #153663
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp documentation technical debt test test-api-integration

Coverage
Beta Give feedback
  1. [Security Solution] Audit test coverage: Detection Engine area #153638
    Team: SecuritySolution Team:Detection Engine Team:Detections and Resp technical debt test test-api-integration test-coverage test_ui_functional
    yctercero
  2. [Security Solution] Audit test coverage: Rule Management area #153637
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test-api-integration test-coverage test_ui_functional
    banderror
  3. [Security Solution] Increase test coverage: Detection Engine area #161508
    0 of 1
    Meta Team: SecuritySolution Team:Detection Engine Team:Detections and Resp technical debt test test-coverage
  4. [Security Solution] Increase test coverage: Rule Management area #161509
    0 of 2
    Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test test-coverage

Maintainability
Beta Give feedback
  1. [Security Solution] Support rule id in wait for rule status helper #153410
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:prev-minor refactoring release_note:skip technical debt test-api-integration v8.7.1 v8.8.0
    maximpn
  2. [Security Solution] Allow rewriting rule create props in Cypress tests #153474
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:prev-minor refactoring release_note:skip technical debt test test_ui_functional v8.7.1 v8.8.0
    maximpn
  3. [Security Solution] Cypress: remove duplicate properties from rule mocks #153694
    Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring technical debt test test_ui_functional

Misc
Beta Give feedback
  1. [Security Solution] Cypress test runner doesn't start locally on Intel-based Macs #164435
    Team: SecuritySolution Team:Detection Engine Team:Detection Rule Management Team:Detections and Resp Team:Threat Hunting bug technical debt test
@banderror banderror added test test_ui_functional test-api-integration test-coverage issues & PRs for improving code test coverage Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Detection Alerts Security Solution Detection Alerts Feature Team:Security Solution Platform Security Solution Platform Team Team:Detection Rule Management Security Detection Rule Management Team Team:Detection Alerts Security Detection Alerts Area Team 8.8 candidate labels Mar 24, 2023
@banderror banderror mentioned this issue Mar 24, 2023
Closed
@banderror banderror added the epic label Mar 24, 2023
This was referenced Mar 24, 2023
Open
Open
Closed
Open
Open
Open
@banderror banderror added the technical debt Improvement of the software architecture and operational architecture label Mar 24, 2023
This was referenced Mar 24, 2023
Open
Open
Open
Open
Open
Open
This was referenced Jul 20, 2023
Open
Merged
banderror added a commit that referenced this issue Jul 25, 2023
@banderror
[Security Solution] Restructure Cypress tests under `security_solutio…
4087316 
…n/cypress/e2e/detection_rules` folder (#162373)

**Epic:** #153633
**Partially addresses:** #153645

## Summary

This PR builds upon #161900 and
moves tests located in the `e2e/detection_rules` folder into
`e2e/detection_response` and splits them into multiple sub-folders
according to the Detection Engine subdomains we have. It also updates
the CODEOWNERS file accordingly.

<img width="451" alt="Screenshot 2023-07-25 at 21 03 08"
src="https://github.com/elastic/kibana/assets/7359339/fb6052c9-3c5d-4547-98f1-61f44b9f7187">

## Details

Specifically, changes in this PR include:

- The `e2e/detections_response` folder was renamed to
`e2e/detection_response`.
- The `e2e/detections_response/bulk_actions` folder became
`e2e/detection_response/rule_management/rule_actions/bulk_actions`.
- Cypress tests for rule types (which actually test rule creation for
different rule types) were moved to
`e2e/detection_response/rule_creation`.
- The CODEOWNERS file was updated.

Things not addressed in this PR:

- No ownership was assigned for `e2e/detection_response/rule_actions`.
Will need to figure this out with @yctercero.
- No restructuring was done for `security_solution/cypress/screens` and
`security_solution/cypress/tasks`. Will be done in follow-up PRs.
- No refactoring was done for the tests themselves. Some of this work is
also upcoming.

The full file structure of the `detection_response` tests looks like
this:

<img width="452" alt="Screenshot 2023-07-25 at 21 03 44"
src="https://github.com/elastic/kibana/assets/7359339/2b89c6d2-9f2d-4cf6-914f-a71c3fa93595">
rshen91 pushed a commit to rshen91/kibana that referenced this issue Jul 26, 2023
@banderror @rshen91
[Security Solution] Restructure Cypress tests under `security_solutio…
2e691ab 
…n/cypress/e2e/detection_rules` folder (elastic#162373)

**Epic:** elastic#153633
**Partially addresses:** elastic#153645

## Summary

This PR builds upon elastic#161900 and
moves tests located in the `e2e/detection_rules` folder into
`e2e/detection_response` and splits them into multiple sub-folders
according to the Detection Engine subdomains we have. It also updates
the CODEOWNERS file accordingly.

<img width="451" alt="Screenshot 2023-07-25 at 21 03 08"
src="https://github.com/elastic/kibana/assets/7359339/fb6052c9-3c5d-4547-98f1-61f44b9f7187">

## Details

Specifically, changes in this PR include:

- The `e2e/detections_response` folder was renamed to
`e2e/detection_response`.
- The `e2e/detections_response/bulk_actions` folder became
`e2e/detection_response/rule_management/rule_actions/bulk_actions`.
- Cypress tests for rule types (which actually test rule creation for
different rule types) were moved to
`e2e/detection_response/rule_creation`.
- The CODEOWNERS file was updated.

Things not addressed in this PR:

- No ownership was assigned for `e2e/detection_response/rule_actions`.
Will need to figure this out with @yctercero.
- No restructuring was done for `security_solution/cypress/screens` and
`security_solution/cypress/tasks`. Will be done in follow-up PRs.
- No refactoring was done for the tests themselves. Some of this work is
also upcoming.

The full file structure of the `detection_response` tests looks like
this:

<img width="452" alt="Screenshot 2023-07-25 at 21 03 44"
src="https://github.com/elastic/kibana/assets/7359339/2b89c6d2-9f2d-4cf6-914f-a71c3fa93595">
ThomThomson pushed a commit to ThomThomson/kibana that referenced this issue Aug 1, 2023
@banderror
[Security Solution] Restructure Cypress tests under `security_solutio…
212b154 
…n/cypress/e2e/detection_rules` folder (elastic#162373)

**Epic:** elastic#153633
**Partially addresses:** elastic#153645

## Summary

This PR builds upon elastic#161900 and
moves tests located in the `e2e/detection_rules` folder into
`e2e/detection_response` and splits them into multiple sub-folders
according to the Detection Engine subdomains we have. It also updates
the CODEOWNERS file accordingly.

<img width="451" alt="Screenshot 2023-07-25 at 21 03 08"
src="https://github.com/elastic/kibana/assets/7359339/fb6052c9-3c5d-4547-98f1-61f44b9f7187">

## Details

Specifically, changes in this PR include:

- The `e2e/detections_response` folder was renamed to
`e2e/detection_response`.
- The `e2e/detections_response/bulk_actions` folder became
`e2e/detection_response/rule_management/rule_actions/bulk_actions`.
- Cypress tests for rule types (which actually test rule creation for
different rule types) were moved to
`e2e/detection_response/rule_creation`.
- The CODEOWNERS file was updated.

Things not addressed in this PR:

- No ownership was assigned for `e2e/detection_response/rule_actions`.
Will need to figure this out with @yctercero.
- No restructuring was done for `security_solution/cypress/screens` and
`security_solution/cypress/tasks`. Will be done in follow-up PRs.
- No refactoring was done for the tests themselves. Some of this work is
also upcoming.

The full file structure of the `detection_response` tests looks like
this:

<img width="452" alt="Screenshot 2023-07-25 at 21 03 44"
src="https://github.com/elastic/kibana/assets/7359339/2b89c6d2-9f2d-4cf6-914f-a71c3fa93595">
This was referenced Aug 22, 2023
Open
Closed
Closed
Open
Open
This was referenced Aug 31, 2023
Closed
Closed
@yctercero yctercero mentioned this issue Sep 5, 2023
Open
3 tasks
@maximpn maximpn mentioned this issue Sep 11, 2023
Open
2 tasks
@yctercero yctercero mentioned this issue Oct 17, 2023
Open
@banderror banderror mentioned this issue Dec 20, 2023
Closed
@banderror banderror mentioned this issue Feb 14, 2024
Open
@banderror banderror mentioned this issue Mar 22, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@banderror banderror

@yctercero yctercero

Labels
epic Feature:Detection Alerts Security Solution Detection Alerts Feature Feature:Detection Rules Anything related to Security Solution's Detection Rules Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration test-coverage issues & PRs for improving code test coverage v8.10.0 v8.11.0 v8.12.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@banderror @yctercero @approksiu @marshallmain