Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Fix flaky tests: Detection and Response team 8.11-8.12 #165359

Closed
9 tasks done
Tracked by #153633
banderror opened this issue Aug 31, 2023 · 6 comments
Closed
9 tasks done
Tracked by #153633

[Security Solution] Fix flaky tests: Detection and Response team 8.11-8.12 #165359

banderror opened this issue Aug 31, 2023 · 6 comments
Assignees
@maximpn @jpdjere
Labels
8.12 candidate Meta Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration v8.11.0 v8.12.0

Comments

@banderror
Copy link
Contributor

banderror commented Aug 31, 2023 •
edited

Epic: #153633

Summary

Fix flaky tests and unskip tests that belong to the Rule Management or the Detection Engine teams.

Our "inbox" of failed and skipped tests:

FTR tests: #165780

Sub-tasks

Prebuilt rules
Beta Give feedback
  1. Failing test: should notify user about prebuilt rules available for installation and for upgrade - Detection rules, Prebuilt Rules Installation and Update Notifications Notifications Rule installation available and rule update available notifications should notify user about prebuilt rules available for installation and for upgrade #163661
    8.11 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/bundled_prebuilt_rules_package/prerelease_packages·ts - prerelease_packages should install latest stable version and ignore prerelease packages #162192
    8.12 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  3. Failing test: should notify user about prebuilt rules package available for update - Detection rules, Prebuilt Rules Installation and Update Notifications Notifications Rule update notification should notify user about prebuilt rules package available for update #163772
    8.12 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/prebuilt_rules/fleet_integration·ts - detection engine api security and spaces enabled - Prebuilt Rules install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage #167056
    8.12 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  5. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows·cy·ts - Detection rules, Prebuilt Rules Installation and Update workflow Installation of prebuilt rules package via Fleet should install rules from the Fleet package when user clicks on CTA should install rules from the Fleet package when user clicks on CTA #168897
    8.12 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere

Rule management and monitoring
Beta Give feedback
  1. [Security Solution] Rules table always refreshes upon switching from another tab/window #165221
    8.11 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp bug v8.11.0
    maximpn
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/get_rule_execution_results·ts - detection engine api security and spaces enabled - Group 10 Get Rule Execution Results should return execution events ordered by @timestamp desc when a status filter is active and there are more than 1000 executions #165129
    8.12 candidate Feature:Rule Monitoring Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere
  3. [Security Solution] Flaky test: Rule Deletion - Rule deletion User can delete multiple selected rules via a bulk action User can delete multiple selected rules via a bulk action #169083
    8.12 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp failed-test
    jpdjere

Related integrations
Beta Give feedback
  1. [Security Solution] Reduce flakiness in Cypress tests for Related Integrations #165504
    8.11 candidate Feature:Related Integrations Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-failure-flaky
    maximpn
@banderror banderror added test Meta test_ui_functional technical debt Improvement of the software architecture and operational architecture test-api-integration Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Team:Detection Engine Security Solution Detection Engine Area 8.11 candidate v8.11.0 labels Aug 31, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

This was referenced Aug 31, 2023
Open
Closed
Closed
@yctercero yctercero mentioned this issue Sep 5, 2023
Open
3 tasks
@maximpn maximpn mentioned this issue Sep 18, 2023
Merged
maximpn added a commit that referenced this issue Sep 22, 2023
@maximpn
[Security Solution] Unskip related integrations Cypress tests in ESS (#…
f52ca02 
…166659)

**Related to: #165359
**Resolves: #165504

## Summary

This PR unskips skipped rule integrations ESS tests.

## Details

Besides just unskipping ESS tests this PR simplifies the test by fetching alert data from the ES directly instead of using multiple steps in UI to validate an expected field has an expected value.

## Flaky test runner

ESS [related_integrations.cy.ts 100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3199) 🟢
@banderror banderror changed the title [Security Solution] Fix flaky tests: Detection and Response team 8.11 [Security Solution] Fix flaky tests: Detection and Response team 8.11-8.12 Oct 13, 2023
@banderror banderror assigned jpdjere and unassigned yctercero and WafaaNasr Oct 16, 2023
@banderror
Copy link
Contributor Author

The tickets related to true flakes will be assigned to the corresponding teams.

@jpdjere
Copy link
Contributor

jpdjere commented Oct 19, 2023 •
edited by banderror

PRs used to triage Detection Engine tests reported in this list, plus their backports (indented):

@jpdjere
Copy link
Contributor

jpdjere commented Oct 30, 2023 •
edited by banderror

Rules management flaky test fix PRs plus their backports (indented):

@banderror
Copy link
Contributor Author

banderror commented Nov 15, 2023 •
edited

We have eliminated test flakiness in the Rule Management area almost entirely, and have greatly reduced it in the Detection Engine area. At the time of writing, there's some leftover flakiness you can track using these links:

@maximpn and @jpdjere: you've done an amazing job, thank you so much for all your efforts made during the last 3 release cycles fighting the flakiness, unskipping tests, and enabling them in Serverless!

I consider the 8.11 and 8.12 goals of this ticket accomplished, and so I'm closing it. Both teams will handle the leftover flakiness as part of separate tickets. cc @yctercero

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

@jpdjere jpdjere

Labels
8.12 candidate Meta Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. technical debt Improvement of the software architecture and operational architecture test_ui_functional test test-api-integration v8.11.0 v8.12.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@maximpn @jpdjere @banderror @yctercero @WafaaNasr @elasticmachine