Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Enable Serverless for exceptions/entry and exceptions/rule_details_flow Cypress tests #169211

Conversation

jpdjere
Copy link
Contributor

@jpdjere jpdjere commented Oct 18, 2023

Summary

Flaky test runner for Cypress tests in:

  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow

Changes

  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts enabled on Serverless
  • x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts removed from Serverless testing

Related failing-test issues

exceptions/entry

  1. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions·cy·ts - Add multiple conditions and validate the generated exceptions "before each" hook for "Use multipe AND conditions and validate it generates one exception" "before each" hook for "Use multipe AND conditions and validate it generates one exception" #165734
  2. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions·cy·ts - Add multiple conditions and validate the generated exceptions Use multipe OR conditions and validate it generates multiple exceptions Use multipe OR conditions and validate it generates multiple exceptions #165652
  3. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions·cy·ts - Add multiple conditions and validate the generated exceptions Use multipe AND conditions and validate it generates one exception Use multipe AND conditions and validate it generates one exception #165651

exceptions/rule_details_flow

  1. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/add_edit_endpoint_exception·cy·ts - Add endpoint exception from rule details "before all" hook for "creates an exception item" "before all" hook for "creates an exception item" #165773
  2. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/add_edit_endpoint_exception·cy·ts - Add endpoint exception from rule details "before each" hook for "creates an exception item" "before each" hook for "creates an exception item" #165742
  3. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/add_edit_endpoint_exception·cy·ts - Add endpoint exception from rule details edits an endpoint exception item edits an endpoint exception item #165736
  4. Failing test: Security Solution Cypress.x-pack/plugins/security_solution/cypress/e2e/exceptions/rule_details_flow/add_edit_exception·cy·ts - Add/edit exception from rule details existing list and items rule with existing shared exceptions Creates an exception item to add to rule only Creates an exception item to add to rule only #163923
  5. Failing test: "before each" hook for "Cannot add an exception from empty viewer screen" - Exceptions viewer read only "before each" hook for "Cannot add an exception from empty viewer screen" #164007
  6. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/add_edit_exception_data_view.cy.ts - /Add exception using data views from rule details -- Creates an exception item and close all matching alerts #169382
    Unreported flaky test that was discovered when investigating these issues. Tagged as legit-flake and assigned to @elastic/security-detection-engine
  7. Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/add_edit_exception_data_view·cy·ts - Add exception using data views from rule details Edits an exception item Edits an exception item #169274
    Flaky test reported by the Kibana machine while this investigation was taking place.
    Since the flaky test runner for the related tests detected flakiness for the same test file (though not same test), I'm marking this as legit-flake for it to be investigated.

Flaky test runner link

@jpdjere jpdjere requested a review from a team as a code owner October 18, 2023 09:15
@jpdjere jpdjere marked this pull request as draft October 19, 2023 09:06
This was referenced Oct 19, 2023
@jpdjere jpdjere self-assigned this Oct 23, 2023
@jpdjere jpdjere changed the title [Security Solution] Flaky test runner for exceptions/entry and exceptions/rule_details_flow Cypress tests [Security Solution] Enable Serverless for exceptions/entry and exceptions/rule_details_flow Cypress tests Oct 23, 2023
@jpdjere jpdjere marked this pull request as ready for review October 23, 2023 20:58
@jpdjere jpdjere added Team:Detections and Resp Security Detection Response Team Team:Detection Engine Security Solution Detection Engine Area v8.12.0 labels Oct 23, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@jpdjere jpdjere added the release_note:skip Skip the PR/issue when compiling release notes label Oct 23, 2023
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Investigations - Security Solution Cypress Tests #5 / Save Timeline Prompts "before each" hook for "When user navigates to the page where timeline is present, Time save modal should not exists." "before each" hook for "When user navigates to the page where timeline is present, Time save modal should not exists."

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jpdjere

@jpdjere jpdjere merged commit 9a73009 into elastic:main Oct 24, 2023
33 checks passed
@jpdjere jpdjere deleted the flaky-test-runner-exceptions-entry-and_rule_details_flow branch October 24, 2023 16:04
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 24, 2023
…eptions/rule_details_flow` Cypress tests (elastic#169211)

## Summary

Flaky test runner for Cypress tests in:
- `x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry`
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow`

## Changes

-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts`
**removed from Serverless testing**

## Related failing-test issues

### `exceptions/entry`
1. elastic#165734
2. elastic#165652
3. elastic#165651

### `exceptions/rule_details_flow`
4. elastic#165773
5. elastic#165742
6. elastic#165736
7. elastic#163923
8. elastic#164007
9. elastic#169382
**Unreported flaky test that was discovered when investigating these
issues. Tagged as `legit-flake` and assigned to
@elastic/security-detection-engine**
10. elastic#169274
**Flaky test reported by the Kibana machine while this investigation was
taking place.
Since the flaky test runner for the related tests detected flakiness for
the same test file (though not same test), I'm marking this as
legit-flake for it to be investigated.**

### Flaky test runner link

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3577
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3621
[SERVERLESS TAGS ADJUSTED]

(cherry picked from commit 9a73009)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

cqliu1 pushed a commit to cqliu1/kibana that referenced this pull request Oct 25, 2023
…eptions/rule_details_flow` Cypress tests (elastic#169211)

## Summary

Flaky test runner for Cypress tests in:
- `x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry` 
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow`

## Changes

-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts`
**enabled on Serverless**
-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts`
**removed from Serverless testing**

## Related failing-test issues

### `exceptions/entry`
1. elastic#165734
2. elastic#165652
3. elastic#165651

### `exceptions/rule_details_flow`
4. elastic#165773
5. elastic#165742
6. elastic#165736
7. elastic#163923
8. elastic#164007
9. elastic#169382
**Unreported flaky test that was discovered when investigating these
issues. Tagged as `legit-flake` and assigned to
@elastic/security-detection-engine**
10. elastic#169274
**Flaky test reported by the Kibana machine while this investigation was
taking place.
Since the flaky test runner for the related tests detected flakiness for
the same test file (though not same test), I'm marking this as
legit-flake for it to be investigated.**

### Flaky test runner link

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3577
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3621
[SERVERLESS TAGS ADJUSTED]
kibanamachine added a commit that referenced this pull request Oct 26, 2023
…nd `exceptions/rule_details_flow` Cypress tests (#169211) (#169679)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution] Enable Serverless for `exceptions/entry` and
`exceptions/rule_details_flow` Cypress tests
(#169211)](#169211)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2023-10-24T16:04:12Z","message":"[Security
Solution] Enable Serverless for `exceptions/entry` and
`exceptions/rule_details_flow` Cypress tests (#169211)\n\n##
Summary\r\n\r\nFlaky test runner for Cypress tests in:\r\n-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry`
\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n## Related failing-test
issues\r\n\r\n### `exceptions/entry`\r\n1.
#165734.
#165652.
#165651
`exceptions/rule_details_flow`\r\n4.
#165773.
#165742.
#165736.
#163923.
#164007.
#169382 flaky
test that was discovered when investigating these\r\nissues. Tagged as
`legit-flake` and assigned
to\r\n@elastic/security-detection-engine**\r\n10.
#169274 test reported
by the Kibana machine while this investigation was\r\ntaking
place.\r\nSince the flaky test runner for the related tests detected
flakiness for\r\nthe same test file (though not same test), I'm marking
this as\r\nlegit-flake for it to be investigated.**\r\n\r\n### Flaky
test runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3577\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3621\r\n[SERVERLESS
TAGS
ADJUSTED]","sha":"9a73009c2b2effab436cae61735cccc79af1a21d","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Detections
and Resp","Team:Detection
Engine","v8.11.0","v8.12.0"],"number":169211,"url":"#169211
Solution] Enable Serverless for `exceptions/entry` and
`exceptions/rule_details_flow` Cypress tests (#169211)\n\n##
Summary\r\n\r\nFlaky test runner for Cypress tests in:\r\n-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry`
\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n## Related failing-test
issues\r\n\r\n### `exceptions/entry`\r\n1.
#165734.
#165652.
#165651
`exceptions/rule_details_flow`\r\n4.
#165773.
#165742.
#165736.
#163923.
#164007.
#169382 flaky
test that was discovered when investigating these\r\nissues. Tagged as
`legit-flake` and assigned
to\r\n@elastic/security-detection-engine**\r\n10.
#169274 test reported
by the Kibana machine while this investigation was\r\ntaking
place.\r\nSince the flaky test runner for the related tests detected
flakiness for\r\nthe same test file (though not same test), I'm marking
this as\r\nlegit-flake for it to be investigated.**\r\n\r\n### Flaky
test runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3577\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3621\r\n[SERVERLESS
TAGS
ADJUSTED]","sha":"9a73009c2b2effab436cae61735cccc79af1a21d"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"#169211
Solution] Enable Serverless for `exceptions/entry` and
`exceptions/rule_details_flow` Cypress tests (#169211)\n\n##
Summary\r\n\r\nFlaky test runner for Cypress tests in:\r\n-
`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry`
\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow`\r\n\r\n##
Changes\r\n\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/use_value_list.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/entry/multiple_conditions.cy.ts`\r\n**enabled
on
Serverless**\r\n-\r\n`x-pack/test/security_solution_cypress/cypress/e2e/exceptions/rule_details_flow/read_only_view.cy.ts`\r\n**removed
from Serverless testing**\r\n\r\n## Related failing-test
issues\r\n\r\n### `exceptions/entry`\r\n1.
#165734.
#165652.
#165651
`exceptions/rule_details_flow`\r\n4.
#165773.
#165742.
#165736.
#163923.
#164007.
#169382 flaky
test that was discovered when investigating these\r\nissues. Tagged as
`legit-flake` and assigned
to\r\n@elastic/security-detection-engine**\r\n10.
#169274 test reported
by the Kibana machine while this investigation was\r\ntaking
place.\r\nSince the flaky test runner for the related tests detected
flakiness for\r\nthe same test file (though not same test), I'm marking
this as\r\nlegit-flake for it to be investigated.**\r\n\r\n### Flaky
test runner
link\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3577\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3621\r\n[SERVERLESS
TAGS ADJUSTED]","sha":"9a73009c2b2effab436cae61735cccc79af1a21d"}}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <jpdjeredjian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team v8.11.0 v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants