Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Unskip and enable for Serverless shared_exception_lists_management Cypress tests #169182

Merged

Conversation

jpdjere
Copy link
Contributor

@jpdjere jpdjere commented Oct 17, 2023

Summary

Running flaky test runner for:

  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management

Changes

  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts unskipped and enabled in Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/read_only.cy.ts removed from Serverless testing

Related failing-test issues

  1. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table·cy·ts - Filter Lists "before each" hook for "Filters exception lists on search" "before each" hook for "Filters exception lists on search" #165874
  2. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists·cy·ts - Manage lists from "Shared Exception Lists" page Create/Export/Delete List Export exception list Export exception list #165838
  3. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions·cy·ts - Add, edit and delete exception Add, Edit and delete Exception item "before each" hook for "should create exception item from Shared Exception List page and linked to a Rule" "before each" hook for "should create exception item from Shared Exception List page and linked to a Rule" #165795
  4. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details·cy·ts - Exception list detail page "before each" hook for "Should edit list details" "before each" hook for "Should edit list details" #165743 - Closed as duplicate of _#165640
  5. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists·cy·ts - Manage lists from "Shared Exception Lists" page Create/Export/Delete List "before each" hook for "Export exception list" "before each" hook for "Export exception list" #165690
  6. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details·cy·ts - Exception list detail page "before all" hook for "Should edit list details" "before all" hook for "Should edit list details" #165640

Flaky test runner link

@jpdjere jpdjere requested a review from a team as a code owner October 18, 2023 07:13
@jpdjere jpdjere changed the title [Security Solution] Flaky test runner for shared_exception_lists_management Cypress tests [Security Solution] Flaky test runner for shared_exception_lists_management Cypress tests Oct 18, 2023
@jpdjere jpdjere marked this pull request as draft October 19, 2023 09:06
This was referenced Oct 19, 2023
Copy link

@TokenPayId TokenPayId left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jpdjere:flaky-run-shared_exception_lists_management

@jpdjere jpdjere marked this pull request as ready for review October 24, 2023 16:12
@jpdjere jpdjere self-assigned this Oct 24, 2023
@jpdjere jpdjere added release_note:fix Team:Detections and Resp Security Detection Response Team Team:Detection Engine Security Solution Detection Engine Area v8.11.0 v8.12.0 labels Oct 24, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@jpdjere jpdjere changed the title [Security Solution] Flaky test runner for shared_exception_lists_management Cypress tests [Security Solution] Unskip and enable for Serverless shared_exception_lists_management Cypress tests Oct 24, 2023
Copy link
Contributor

@yctercero yctercero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but running flakey test runner one more time as that last link in the description showed failures so just want to be sure - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3828

@jpdjere jpdjere force-pushed the flaky-run-shared_exception_lists_management branch from cbb51c7 to 48b82a6 Compare November 2, 2023 15:37
@jpdjere
Copy link
Contributor Author

jpdjere commented Nov 3, 2023

  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts whole file broken in serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts Exception list detail page - Should create a new list and link it to two rules -> flaky in both ESS and serverless

-x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/duplicate_lists.cy.ts -> Duplicate List - Duplicate exception list without expired items -> running flaky test runner again to try to reproduce flake

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Serverless Osquery Cypress Tests #1 / App Features for Security Complete PLI response actions should not be available response actions should not be available

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere merged commit c0c7d13 into elastic:main Nov 3, 2023
26 checks passed
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.11 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.11:
- [Security Solution] fix cypress config to run all tests (#169942)
- [Security Solution] Adding serverlessQA tag (#167494)

Manual backport

To create the backport manually run:

node scripts/backport --pr 169182

Questions ?

Please refer to the Backport tool documentation

@banderror
Copy link
Contributor

@jpdjere The backport has failed, would you mind backporting manually? Please make sure to not backport the erroneously merged changes to the package.json file.

jpdjere added a commit to jpdjere/kibana that referenced this pull request Nov 6, 2023
…n_lists_management` Cypress tests (elastic#169182)

## Summary

Running flaky test runner for:
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management`

## Changes

-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts`
**unskipped and enabled in Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/read_only.cy.ts`
**removed from Serverless testing**

## Related failing-test issues
1. elastic#165874
2. elastic#165838
3. elastic#165795
4. elastic#165743 - **Closed as
duplicate of _#165640**
5. elastic#165690
6. elastic#165640

### Flaky test runner link

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3557#_
(Only ESS) 🟢
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3564#_
(ESS and Serverless)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3613
(Serverless evaluated and corrected)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3878
[V3]
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3891
[V4]

(cherry picked from commit c0c7d13)

# Conflicts:
#	x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts
#	x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts
#	x-pack/test/security_solution_cypress/package.json
@jpdjere
Copy link
Contributor Author

jpdjere commented Nov 6, 2023

💚 All backports created successfully

Status Branch Result
8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

banderror added a commit that referenced this pull request Nov 6, 2023
…oductivity team (#170662)

## Summary

This PR makes the @elastic/security-engineering-productivity team the
owner of the `x-pack/test/security_solution_cypress/package.json` file
to prevent accidentally merging temporary changes to this file like in
#169182.

Also, it simplifies the ownership rules a bit. The following lines
assign the team to all files under the two specified folders, but not
files inside their subfolders, so you don't have to specify individual
files:

```
/x-pack/test/security_solution_cypress/* @elastic/security-engineering-productivity
/x-pack/test/security_solution_cypress/cypress/* @elastic/security-engineering-productivity
```
jpdjere added a commit that referenced this pull request Nov 6, 2023
…xception_lists_management` Cypress tests (#169182) (#170669)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution] Unskip and enable for Serverless
`shared_exception_lists_management` Cypress tests
(#169182)](#169182)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2023-11-03T22:35:09Z","message":"[Security
Solution] Unskip and enable for Serverless
`shared_exception_lists_management` Cypress tests (#169182)\n\n##
Summary\r\n\r\nRunning flaky test runner
for:\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts`\r\n**unskipped
and enabled in
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/read_only.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n\r\n## Related failing-test
issues\r\n1. #165874.
#165838.
#165795.
#165743 - **Closed
as\r\nduplicate of _#165640**\r\n5.
#165690.
#165640 Flaky test
runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3557#_\r\n(Only
ESS)
🟢\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3564#_\r\n(ESS
and
Serverless)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3613\r\n(Serverless
evaluated and
corrected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3878\r\n[V3]\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3891\r\n[V4]","sha":"c0c7d1365f9e034128de8cbb03ac5733aefcd22f","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Detections
and Resp","Team:Detection
Engine","v8.11.0","v8.12.0"],"number":169182,"url":"#169182
Solution] Unskip and enable for Serverless
`shared_exception_lists_management` Cypress tests (#169182)\n\n##
Summary\r\n\r\nRunning flaky test runner
for:\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts`\r\n**unskipped
and enabled in
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/read_only.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n\r\n## Related failing-test
issues\r\n1. #165874.
#165838.
#165795.
#165743 - **Closed
as\r\nduplicate of _#165640**\r\n5.
#165690.
#165640 Flaky test
runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3557#_\r\n(Only
ESS)
🟢\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3564#_\r\n(ESS
and
Serverless)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3613\r\n(Serverless
evaluated and
corrected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3878\r\n[V3]\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3891\r\n[V4]","sha":"c0c7d1365f9e034128de8cbb03ac5733aefcd22f"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"#169182
Solution] Unskip and enable for Serverless
`shared_exception_lists_management` Cypress tests (#169182)\n\n##
Summary\r\n\r\nRunning flaky test runner
for:\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/list_detail_page/list_details.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/manage_exceptions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/filter_table.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/import_lists.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/manage_lists.cy.ts`\r\n**unskipped
and enabled in
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/shared_exception_lists_management/shared_exception_list_page/read_only.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n\r\n## Related failing-test
issues\r\n1. #165874.
#165838.
#165795.
#165743 - **Closed
as\r\nduplicate of _#165640**\r\n5.
#165690.
#165640 Flaky test
runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3557#_\r\n(Only
ESS)
🟢\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3564#_\r\n(ESS
and
Serverless)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3613\r\n(Serverless
evaluated and
corrected)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3878\r\n[V3]\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3891\r\n[V4]","sha":"c0c7d1365f9e034128de8cbb03ac5733aefcd22f"}}]}]
BACKPORT-->
@banderror banderror added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Nov 7, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror banderror added test release_note:skip Skip the PR/issue when compiling release notes and removed release_note:fix labels Nov 7, 2023
jpdjere added a commit that referenced this pull request Nov 8, 2023
…hange (#170663)

## Summary

- Reverts unintentional change to
`x-pack/test/security_solution_cypress/package.json` introduced in
#169182

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test v8.11.0 v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants