Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Unskip tests for Detection Engine and Security Solution API Integration tests #169262

Merged
merged 3 commits into from
Oct 31, 2023

Conversation

jpdjere
Copy link
Contributor

@jpdjere jpdjere commented Oct 18, 2023

Summary

Changes

  • x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts unskipped
  • x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts unskipped

Related flaky test issues

Tests already moved to x-pack/test/security_solution_api_integration

  1. Failing test: Detection Engine Serverless API Integration Tests.x-pack/test/security_solution_api_integration/test_suites/detections_response/exceptions/rule_exception/create_rule_exceptions·ts - Detections response API Exceptions API @serverless @ess create_rule_exception_route creates and associates a rule_default exception list to a rule if one not already found #169055
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group9/exception_operators_data_types/text_array·ts - detection engine api security and spaces enabled - Group 9 Detection exceptions data types and operators Rule exception operators for data type text "is" operator should filter 1 single text if it is set as an exception #168775
    This issue is reported for the legacy test, but is now moved to x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group8/exception_operators_data_types/keyword·ts - detection engine api security and spaces enabled - Group 8 Detection exceptions data types and operators Rule exception operators for data type keyword "is" operator should filter 1 single keyword if it is set as an exception #163511
    This issue is reported for the legacy test, but is now moved to x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts

Test still in x-pack/test/detection_engine_api_integration

  1. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group10/risk_engine/risk_scoring_task_execution·ts - detection engine api security and spaces enabled - Group 10 Risk Engine - Risk Scoring Task with auditbeat data with some alerts containing hosts initializing the risk engine "before each" hook for "calculates and persists risk scores for alert documents" #168415
  2. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule task telemetry pre-built/immutable/elastic rules should show detection_rules telemetry data for each list type should return mutating types such as "id", "@timestamp", etc... for list of type "detection" #164318
  3. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule task telemetry pre-built/immutable/elastic rules should show detection_rules telemetry data for each list type should return mutating types such as "id", "@timestamp", etc... for list of type "detection" #164313
  4. Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning·ts - detection engine api security and spaces enabled - rule execution logic Machine learning type rules "before all" hook for "should create 1 alert from ML rule when record meets anomaly_threshold" #145776

Other

  1. Failing test: X-Pack API Integration Tests.x-pack/test/api_integration/apis/lists/create_exception_list_item·ts - Lists plugin Lists API "before all" hook for "should return a 400 if an endpoint exception item with a list-based entry is provided" #151636

Flaky test runner link

Part 1 - x-pack/test/detection_engine_api_integration - tests not yet migrated
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597
Part 2 - x-pack/test/security_solution_api_integration - tests already moved
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

@jpdjere jpdjere requested a review from a team as a code owner October 18, 2023 15:36
@jpdjere jpdjere marked this pull request as draft October 19, 2023 09:09
This was referenced Oct 19, 2023
@jpdjere jpdjere changed the title [Security Solution] Flaky test runner for Detection Engine API Integration tests [Security Solution] Flaky test runner for Detection Engine and Security Solution API Integration tests Oct 19, 2023
@jpdjere jpdjere self-assigned this Oct 24, 2023
@jpdjere jpdjere changed the title [Security Solution] Flaky test runner for Detection Engine and Security Solution API Integration tests [Security Solution] Unskip tests for Detection Engine and Security Solution API Integration tests Oct 24, 2023
@jpdjere jpdjere added release_note:fix Team:Detections and Resp Security Detection Response Team Team:Detection Engine Security Solution Detection Engine Area v8.11.0 v8.12.0 labels Oct 24, 2023
@jpdjere jpdjere marked this pull request as ready for review October 24, 2023 16:24
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere requested a review from a team October 25, 2023 09:18
@jpdjere jpdjere merged commit 9eff64a into elastic:main Oct 31, 2023
46 checks passed
@jpdjere jpdjere deleted the flaky-test-integration-tests-security branch October 31, 2023 10:36
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 31, 2023
…lution API Integration tests (elastic#169262)

## Summary

### Changes

-
`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`
**unskipped**
-
`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`
**unskipped**

### Related flaky test issues

**Tests already moved to
`x-pack/test/security_solution_api_integration`**

1. elastic#169055
2. elastic#168775
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`
3. elastic#163511
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`

- Flaky test runner for above tests:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

**Test still in `x-pack/test/detection_engine_api_integration`**
1. elastic#168415
2. elastic#164318
3. elastic#164313
4. elastic#145776

**Other**
1. elastic#151636

## Flaky test runner link

Part 1 - `x-pack/test/detection_engine_api_integration` - tests not yet
migrated
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597
Part 2 - `x-pack/test/security_solution_api_integration` - tests already
moved
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

(cherry picked from commit 9eff64a)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

jpdjere added a commit that referenced this pull request Oct 31, 2023
…rity Solution API Integration tests (#169262) (#170218)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution] Unskip tests for Detection Engine and Security
Solution API Integration tests
(#169262)](#169262)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2023-10-31T10:36:41Z","message":"[Security
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
#169055.
#168775 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
#163511 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
#168415.
#164318.
#164313.
#145776.
#151636 Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Detections
and Resp","Team:Detection
Engine","v8.11.0","v8.12.0"],"number":169262,"url":"#169262
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
#169055.
#168775 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
#163511 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
#168415.
#164318.
#164313.
#145776.
#151636 Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"#169262
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
#169055.
#168775 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
#163511 issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
#168415.
#164318.
#164313.
#145776.
#151636 Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999"}}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <jpdjeredjian@gmail.com>
awahab07 pushed a commit to awahab07/kibana that referenced this pull request Oct 31, 2023
…lution API Integration tests (elastic#169262)

## Summary

### Changes

-
`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`
**unskipped**
-
`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`
**unskipped**

### Related flaky test issues

**Tests already moved to
`x-pack/test/security_solution_api_integration`**

1. elastic#169055
2. elastic#168775
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`
3. elastic#163511
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`

- Flaky test runner for above tests:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

**Test still in `x-pack/test/detection_engine_api_integration`**
1. elastic#168415
2. elastic#164318
3. elastic#164313
4. elastic#145776

**Other**
1. elastic#151636

## Flaky test runner link

Part 1 - `x-pack/test/detection_engine_api_integration` - tests not yet
migrated
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597
Part 2 - `x-pack/test/security_solution_api_integration` - tests already
moved
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:fix Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team v8.11.0 v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants