[Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902

xcrzx · 2023-02-22T16:40:34Z

Summary

The current grouping of integration tests in the Security Solution is difficult for developers to understand. The test groups are simply numbered from 1 to 10 without any logical structure. Consequently, it is challenging to know where to place new tests, and they end up being randomly assigned to groups.

To address this issue, we need to take the following steps:

Determine the current limit for parallel group executions and increase it if necessary.
Restructure the groups by introducing feature-centric names such as prebuilt_rules, rule_management, etc. These groups should be registered in CI in .buildkite/ftr_configs.yml.
Establish code ownership so that teams are responsible for the groups they own.

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-02-22T16:40:37Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2023-02-22T16:40:41Z

Pinging @elastic/security-solution (Team: SecuritySolution)

WafaaNasr · 2023-10-11T10:47:52Z

Tracking the progress in this document

…lastic#170350) ## Summary - Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for Alerts - Introduced a new folder called `alerts` under `security_solution_api_integration` - Moved the utility files associated with alerts to the new directory `security_solution_api_integration`. Files that were not actively used in the previous folder were moved, while any duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old / new groups details and execution time [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Delete| group1/find_rule_exception_references.ts| Already moved in previous PR | | Delete|security_and_spaces/group6 | - | | Move|detection_engine_api_integration/security_and_spaces/group6 |detections_response/default_license/alerts/alerts_compatibility.ts| | Move|detection_engine_api_integration/security_and_spaces/group1| detections_response/default_license/alerts/aliases.ts | | Move |detection_engine_api_integration/security_and_spaces/group1| detections_response/default_license/alerts/create_index.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/create_signals_migrations| detections_response/default_license/alerts/migrations/create_alerts_migrations.ts | | Move|detection_engine_api_integration/security_and_spaces/group10/delete_signals_migrations| detections_response/default_license/alerts/migrations/delete_alerts_migrations.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations| detections_response/default_license/alerts/migrations/finalize_alerts_migrations.ts | | Move|detection_engine_api_integration/security_and_spaces/group10/get_signals_migration_status| detections_response/default_license/alerts/migrations/get_alerts_migration_status.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/open_close_signals| detections_response/default_license/alerts/open_close_alerts.ts| | Move|detection_engine_api_integration/security_and_spaces/group10/alert_tags| detections_response/default_license/alerts/alert_tags.ts| Note: these tests are skipped on the main branch https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L215 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/open_close_signals.ts#L252 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/finalize_signals_migrations.ts#L192 https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group1/create_index.ts#L42 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…sts (#170530) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Entity Analytics/Risk Engine - Introduced new folders called `entity_analytics/risk_engine` under `security_solution_api_integration` - Moved the utility files associated with risk_engine to the new directory `security_solution_api_integration`. Files that were not actively used in the previous folder were moved, while any duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old / new groups details and execution time [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Moved| group10/risk_engine/init_and_status_apis|/entity_analytics/default_license/risk_engine/init_and_status_apis.ts | | Moved| group10/risk_engine/risk_score_calculation|/entity_analytics/default_license/risk_engine/risk_score_calculation.ts | | Moved| group10/risk_engine/risk_score_preview|/entity_analytics/default_license/risk_engine/risk_score_preview.ts | | Moved| group10/risk_engine/risk_scoring_task_execution|/entity_analytics/default_license/risk_engine/risk_scoring_task_execution.ts | | Moved| group10/risk_engine/telemetry_usage|/entity_analytics/default_license/risk_engine/telemetry_usage.ts | | Moved| group10/risk_engine/utils|entity_analytics/utils/risk_engine.ts | | Moved| utils/get_stats|entity_analytics/utils/get_risk_engine_stats.ts | Tests skipped on main: https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/risk_engine/init_and_status_apis.ts#L363 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nikita Khristinin <nkhristinin@gmail.com>

…sts (elastic#170530) ## Summary Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for Entity Analytics/Risk Engine - Introduced new folders called `entity_analytics/risk_engine` under `security_solution_api_integration` - Moved the utility files associated with risk_engine to the new directory `security_solution_api_integration`. Files that were not actively used in the previous folder were moved, while any duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old / new groups details and execution time [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Moved| group10/risk_engine/init_and_status_apis|/entity_analytics/default_license/risk_engine/init_and_status_apis.ts | | Moved| group10/risk_engine/risk_score_calculation|/entity_analytics/default_license/risk_engine/risk_score_calculation.ts | | Moved| group10/risk_engine/risk_score_preview|/entity_analytics/default_license/risk_engine/risk_score_preview.ts | | Moved| group10/risk_engine/risk_scoring_task_execution|/entity_analytics/default_license/risk_engine/risk_scoring_task_execution.ts | | Moved| group10/risk_engine/telemetry_usage|/entity_analytics/default_license/risk_engine/telemetry_usage.ts | | Moved| group10/risk_engine/utils|entity_analytics/utils/risk_engine.ts | | Moved| utils/get_stats|entity_analytics/utils/get_risk_engine_stats.ts | Tests skipped on main: https://github.com/elastic/kibana/blob/main/x-pack/test/detection_engine_api_integration/security_and_spaces/group10/risk_engine/init_and_status_apis.ts#L363 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nikita Khristinin <nkhristinin@gmail.com>

…security_solution_api_integration` folder (#169951) Addresses partially: #151902 ## Summary - Migrates Prebuilt rules-related API integration tests to `security_solution_api_integration` folder. - Moves tests from `x-pack/test/detection_engine_api_integration/security_and_spaces` into `x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license`. - PR moves following folders: - `/prebuilt_rules` - `/bundled_prebuilt_rules_package` - `/large_prebuilt_rules_package` - `/update_prebuilt_rules_package` - Duplicates or completely moves needed utils into: - `x-pack/test/security_solution_api_integration/test_suites/detections_response/utils/rules/prebuilt_rules`, depending if they are no longer used in the original folder or they still are in the remaining test (should be moved shortly as well) - All tests run on both **ESS** and **Serverless** ## Flaky test runner - [ ] [`/prebuilt_rules`](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3888) 🟢 - [ ] [`/bundled_prebuilt_rules_package`](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3916)🟢 - [ ] [`/large_prebuilt_rules_package`](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3917)🟢 - [ ] [`/update_prebuilt_rules_package`](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3920)🟢 [Link to all for PR](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds?branch=refs%2Fpull%2F169951%2Fhead) ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… logic (#170765) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for rule execution logic - Moved the utility files associated with rule execution logic to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this[document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - Added new `Alert` archive for version `8.8.0` - Resolved the issue with the `query.ts` test where the execution logic is executed last, encompassing the "query" test because it was unloading the alerts document and led to failures in subsequent tests. - For `Alert As Data` in **Serverless** the alert ancestor will be a data-stream however in **ESS** will be `.internal.alerts-security.alerts-default-000001'` | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/rule_execution_logic| - | | Delete|security_and_spaces/group5 | - | | Move|detection_engine_api_integration/security_and_spaces/group5|detections_response/default_license/rule_execution_logic/keyword_family| | Move|detection_engine_api_integration/security_and_spaces/rule_execution_logic| detections_response/default_license/rule_execution_logic/execution_logic | | Move |detection_engine_api_integration/security_and_spaces/group1/ignore_fields| detections_response/default_license/rule_execution_logic/ignore_fields.ts| | Move|detection_engine_api_integration/security_and_spaces/group1/runtime| detections_response/default_license/rule_execution_logic/runtime.ts | | Move|detection_engine_api_integration/security_and_spaces/group1/timestamps| detections_response/default_license/rule_execution_logic/timestamps.ts| --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…ated tests and Action Throttle (#171397) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Privileges related tests and Action Throttle - Moved the utility files associated with telemetry to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - The **Privileges** Tests are skipped in Serverless now until the FTR Roles [PR](#170131) gets merged | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/group4| - | | Move|detection_engine_api_integration/security_and_spaces/group10/throttle|detections_response/default_license/action/throttle| | Move|detection_engine_api_integration/security_and_spaces/group1/check_privileges|detections_response/default_license/actions/check_privileges| | Move|detection_engine_api_integration/security_and_spaces/group10/read_privileges|detections_response/default_license/user_roles/read_privileges|

#171159) ## Summary Following the initial work in this #166755 - Addresses part of #151902 for Telemetry - Moved the utility files associated with telemetry to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) | Action | File | New Path if moved | |--------|------|----------| | Delete| security_and_spaces/group4| - | | Move|detection_engine_api_integration/security_and_spaces/group4|detections_response/default_license/telemetry/keyword_family|

…n engine tests (elastic#171531) ## Summary Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for basic detection engine-related tests - Introduced a new license folder to hold the `Basic` Ess tests and the `Essentials` Serverless tests, is called `basic_essentials_license` - Added new base configurations files for `serverless/config.base.essentials` and `ess/config.base.basic` - Moved the utility files associated with Basic tests to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the CodeOwner file for the newly moved tests - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) - The **Privileges** Tests are skipped in Serverless now until the FTR Roles [PR](elastic#170131) gets merged | Action | File | New Path | |--------|------|----------| | Moved|basic/create_rules| basic_essentials_license/detection_engine/rules/create_rules| | Moved|basic/create_rules| basic_essentials_license/detection_engine/rules/create_ml_rules_privileges| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/open_close_alerts| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/query_alerts_backword_compatibility| | Moved|basic/create_rules| basic_essentials_license/detection_engine/alerts/query_alerts|

…astic#171992) ## Summary Following the initial work in this elastic#166755 - Addresses part of elastic#151902 for List APIs tests - Added a new folder under the `security_solution_api_integration` called `lists_and_exception_lists` to hold the lists and exception lists tests, and split the `List` APIs into two groups since the execution time in Serverless was close to 30 mins - Modified the [x-pack/plugins/lists/common/schemas/response/exception_list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-4e3545fdeb8c8d9467cfa1c4aa88194e189193a92fa6f1cf5f859b1ef1beb45c), [x-pack/plugins/lists/common/schemas/response/exception_list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-c3cc18faf07aab86e307185d41599c3596a3f8b360d3e4829591afa148283238) , [x-pack/plugins/lists/common/schemas/response/list_item_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-608579ca5e65da74f41319a58d81ab12cc3d79d389b087806c7b74949fbc6cc3), [x-pack/plugins/lists/common/schemas/response/list_schema.mock.ts](https://github.com/elastic/kibana/pull/171992/files#diff-efc64eb35937a8da28fc982c527253c0923650ae4163d4bbc203d3ebc2949835) to accept `elastic user` input because it changes in ESS and Serverless - Deleted the `x-pack/test/lists_api_integration` folder - Moved the utility files associated with Basic tests to the new directory `security_solution_api_integration`. Files not actively used in the previous folder were moved, while duplicate files remained in their original positions. - Updated the below files imports from the old `lists_api_integration` folder to the new `lists_and_exception_lists` ``` lists_api_integration/ - exceptions/operators_data_types/date_numeric_types/date.ts - exceptions/operators_data_types/date_numeric_types/double.ts - exceptions/operators_data_types/date_numeric_types/float.ts - exceptions/operators_data_types/date_numeric_types/integer.ts - exceptions/operators_data_types/ips/ip.ts - exceptions/operators_data_types/ips/ip_array.ts - exceptions/operators_data_types/keyword/keyword.ts - exceptions/operators_data_types/keyword/keyword_array.ts - exceptions/operators_data_types/long/long.ts - exceptions/operators_data_types/text/text.ts - exceptions/operators_data_types/text/text_array.ts - exceptions/workflows/create_endpoint_exceptions.ts - exceptions/workflows/create_rule_exceptions.ts - exceptions/workflows/find_rule_exception_references.ts - exceptions/workflows/role_based_add_edit_comments.ts - exceptions/workflows/role_based_rule_exceptions_workflows.ts - exceptions/workflows/rule_exception_synchronizations.ts - rule_execution_logic/execution_logic/esql.ts - rule_execution_logic/execution_logic/machine_learning.ts - rule_execution_logic/execution_logic/new_terms.ts - rule_execution_logic/execution_logic/query.ts - telemetry/task_based/all_types.ts - telemetry/task_based/detection_rules.ts - telemetry/task_based/security_lists.ts ``` ``` These files should be moved too soon to the new `lists_api_integration` detection_engine_api_integration/security_and_spaces/group10 - import_export_rules.ts - import_rules.ts - perform_bulk_actions ``` - Updated the below files imports to the `ftr_provider_context_with_spaces.d.ts` ` - risk_engine/risk_scoring_task/task_execution_nondefault_spaces.ts` - The QA phase concluded with all tests passing successfully. 🟢 - Updated the CodeOwner file for the newly moved tests - Add a new util file to `deleteAllExceptions` under the old `detection_engine_api` folder since the Rule management related-tests are still need to be moved over to the new folder - Old/new group details, decisions, and execution time are mentioned in this [document](https://docs.google.com/document/d/1CRFfDWMzw3ob03euWIvT4-IoiLXjoiPWI8mTBqP4Zks/edit) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

banderror · 2023-12-20T14:25:04Z

Yara and Wafaa have done 99% of the work for this ticket so far, and @yctercero has an in-progress PR that should address the rest. Just updated the assignees to reflect that.

**Resolves: #151902 ## Summary After this PR, all D&R FTRs are moved to new folder where they can be run in ESS and serverless. Please see below table for a summary of what tests need revisiting by the teams. During the test migration there may have been some tests that failed on serverless, but not ESS. Some we were able to fix and get running on both, others are still marked as `brokenInServerless` and need triage.

…5837) **Resolves: elastic#151902 ## Summary After this PR, all D&R FTRs are moved to new folder where they can be run in ESS and serverless. Please see below table for a summary of what tests need revisiting by the teams. During the test migration there may have been some tests that failed on serverless, but not ESS. Some we were able to fix and get running on both, others are still marked as `brokenInServerless` and need triage.

xcrzx mentioned this issue Feb 22, 2023

[Security Solution] Revisit prebuilt detection rules tests #149502

Merged

3 tasks

banderror changed the title ~~[Security Solution] Restructure integrational test groups that run in parallel~~ [Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI Mar 23, 2023

banderror added the test-coverage issues & PRs for improving code test coverage label Mar 23, 2023

banderror mentioned this issue Mar 24, 2023

[Security Solution] Detection Engine Test Automation and Coverage #153633

Open

banderror added test refactoring test-api-integration and removed test-coverage issues & PRs for improving code test coverage labels Mar 24, 2023

banderror self-assigned this Mar 24, 2023

banderror added 8.9 candidate and removed 8.8 candidate labels May 16, 2023

banderror removed their assignment May 16, 2023

banderror self-assigned this May 23, 2023

banderror added 8.10 candidate and removed 8.9 candidate labels Jul 6, 2023

banderror assigned yctercero Jul 8, 2023

banderror added 8.11 candidate and removed 8.10 candidate labels Aug 22, 2023

banderror unassigned banderror and yctercero Aug 22, 2023

banderror assigned WafaaNasr Aug 31, 2023

banderror mentioned this issue Oct 4, 2023

[Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging #166755

Merged

jpdjere mentioned this issue Nov 7, 2023

[Security Solution] Migrate Prebuilt rules API integration tests to security_solution_api_integration folder #169951

Merged

5 tasks

WafaaNasr mentioned this issue Nov 14, 2023

[Security Solution][API testing] Move and restructures Telemetry logic #171159

Merged

banderror assigned banderror and unassigned jpdjere Nov 15, 2023

WafaaNasr mentioned this issue Nov 16, 2023

[Security Solution][API testing] Move and restructures Privileges related tests and Action Throttle #171397

Merged

WafaaNasr mentioned this issue Nov 20, 2023

[Security Solution][API testing] Move and restructures Basic detection engine tests #171531

Merged

WafaaNasr mentioned this issue Nov 27, 2023

[Security Solution][API testing] Move and restructures Lists APIS #171992

Merged

WafaaNasr mentioned this issue Nov 29, 2023

[Security Solution][API testing][Rule Management] Move and restructures remaining rule managements trail-tests #172173

Merged

yctercero mentioned this issue Nov 29, 2023

[Detections Response][FTR] Move remaining basic license FTRs to new folder #172132

Closed

banderror added 8.13 candidate and removed 8.12 candidate labels Dec 20, 2023

banderror assigned yctercero and unassigned banderror Dec 20, 2023

yctercero mentioned this issue Jan 29, 2024

[Detections Response] Finish moving remaining legacy FTRs #175837

Merged

yctercero added this to the D&R 8.13 milestone Feb 12, 2024

yctercero closed this as completed in #175837 Feb 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902

[Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902

xcrzx commented Feb 22, 2023

elasticmachine commented Feb 22, 2023

elasticmachine commented Feb 22, 2023

WafaaNasr commented Oct 11, 2023 •

edited

banderror commented Dec 20, 2023

[Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902

[Security Solution] Restructure API integration tests into meaningful groups that run in parallel on CI #151902

Comments

xcrzx commented Feb 22, 2023

Summary

elasticmachine commented Feb 22, 2023

elasticmachine commented Feb 22, 2023

WafaaNasr commented Oct 11, 2023 • edited

banderror commented Dec 20, 2023

WafaaNasr commented Oct 11, 2023 •

edited