Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Skip all existing Cypress tests in Serverless: Detection & Response area #164441

Closed
3 tasks
Tracked by #153633
banderror opened this issue Aug 22, 2023 · 0 comments · Fixed by #165966
Closed
3 tasks
Tracked by #153633

[Security Solution] Skip all existing Cypress tests in Serverless: Detection & Response area #164441

banderror opened this issue Aug 22, 2023 · 0 comments · Fixed by #165966
Assignees
@banderror
Labels
8.11 candidate Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test

Comments

@banderror
Copy link
Contributor

Epic: #153633

Summary

In #162698 some of the existing Cypress tests owned by @elastic/security-detections-response were included in the Serverless test suite by tagging them with the SERVERLESS tag. Some of the other tests were tagged as BROKEN_IN_SERVERLESS:

kibana/x-pack/test/security_solution_cypress/cypress/tags.ts

Lines 8 to 12 in 546a3cb

export const tag = {
SERVERLESS: '@serverless',
ESS: '@ess',
BROKEN_IN_SERVERLESS: '@brokenInServerless',
};

Problems we have now:

  • We're not sure if the tests that are only tagged as ESS shouldn't ever be enabled in Serverless, or were not tagged as SERVERLESS temporarily.
  • Many tests were tagged as SERVERLESS, but we don't know how flaky they are.

Together with @yctercero, we discussed that we'd like to add tests in the Serverless suite gradually and in a more controlled fashion, such as every area team adds a test in the Serverless suite only when they know for sure it:

  • works in Serverless
  • is not flaky in Serverless

Todo

  • Add a new tag SKIP_IN_SERVERLESS
  • Exclude tests that are marked with SKIP_IN_SERVERLESS in the two serverless cypress configs (local and CI)
  • Add this tag to all existing Cypress tests that are owned by our teams, except those tests that are already tagged as BROKEN_IN_SERVERLESS

As part of follow-up tickets, start gradually including Cypress tests to the Serverless suite.
@banderror banderror added test Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Team:Detection Engine Security Solution Detection Engine Area 8.11 candidate labels Aug 22, 2023
@banderror banderror mentioned this issue Aug 22, 2023
Open
@banderror banderror changed the title [Security Solution] Skip all existing Detection & Response tests in Serverless [Security Solution] Skip all existing Detection & Response Cypress tests in Serverless Aug 22, 2023
@banderror banderror changed the title [Security Solution] Skip all existing Detection & Response Cypress tests in Serverless [Security Solution] Skip all existing Cypress tests in Serverless: Detection & Response area Aug 22, 2023
This was referenced Aug 22, 2023
Closed
Closed
@banderror banderror mentioned this issue Sep 1, 2023
Merged
@banderror banderror assigned banderror and unassigned maximpn Sep 7, 2023
@banderror banderror mentioned this issue Sep 7, 2023
Merged
banderror added a commit that referenced this issue Sep 9, 2023
@banderror
[Security Solution] Skip all Detection & Response Cypress tests in Se…
1d65e78 
…rverless (#165966)

**Resolves: #164441

## Summary

Skips all Cypress tests owned by
@elastic/security-detection-rule-management and
@elastic/security-detection-engine teams in Serverless using the new
`@skipInServerless` tag.

- Adds a new `@skipInServerless` tag
- Updates `x-pack/test/security_solution_cypress/cypress/README.md` to
explain when to use what tags
- Explicitly adds missing tags to all D&R tests
- Adds `// TODO:` comments to tests with links to follow-up tickets
- Fixes the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script (see below)

Follow-up work:

- #161540
- #161539

## Context

> Serverless test failures will soon block PR merge

> During the development of the serverless test infrastructure, we
decided that serverless tests will only soft-fail. That means you see
the test failure in your PR but you're still able to merge. We did that
mainly in order to not block delivery of stateful features and bug fixes
while serverless tests and pipelines were implemented and stabilized.
We now have the major building blocks for the serverless test
infrastructure in place and will integrate serverless tests in our
regular pipelines. As part of this process, we're skipping failing and
flaky serverless tests that came in during the soft-fail phase. Once
this is done, a PR with serverless test failures can no longer be
merged, similar to how we have it for stateful test failures.

> We plan to merge this in the next few days and we'll send out another
notification when it's done.

## Fixing `parallel.ts`

There were two problems with the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script we use for running Cypress tests:

- The script was broken in #162673
(here on [this
line](https://github.com/elastic/kibana/pull/162673/files#diff-9f40ced6d29c4fc2709af881680400293d8ce1bc9ebb07b9138d6d99c83c09c9R67))
- I think it has never supported situations when all tests matching a
spec pattern (such as
`./cypress/e2e/!(investigations|explore)/**/*.cy.ts`) end up being
skipped via Cypress tags (such as `@skipInServerless`)

Both the issues are fixed in this PR.

Code owners are added for this script in the CODEOWNERS file to prevent
breaking this script in future PRs.
kderusso pushed a commit to kderusso/kibana that referenced this issue Sep 11, 2023
@banderror @kderusso
[Security Solution] Skip all Detection & Response Cypress tests in Se…
b1aa3a7 
…rverless (elastic#165966)

**Resolves: elastic#164441

## Summary

Skips all Cypress tests owned by
@elastic/security-detection-rule-management and
@elastic/security-detection-engine teams in Serverless using the new
`@skipInServerless` tag.

- Adds a new `@skipInServerless` tag
- Updates `x-pack/test/security_solution_cypress/cypress/README.md` to
explain when to use what tags
- Explicitly adds missing tags to all D&R tests
- Adds `// TODO:` comments to tests with links to follow-up tickets
- Fixes the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script (see below)

Follow-up work:

- elastic#161540
- elastic#161539

## Context

> Serverless test failures will soon block PR merge

> During the development of the serverless test infrastructure, we
decided that serverless tests will only soft-fail. That means you see
the test failure in your PR but you're still able to merge. We did that
mainly in order to not block delivery of stateful features and bug fixes
while serverless tests and pipelines were implemented and stabilized.
We now have the major building blocks for the serverless test
infrastructure in place and will integrate serverless tests in our
regular pipelines. As part of this process, we're skipping failing and
flaky serverless tests that came in during the soft-fail phase. Once
this is done, a PR with serverless test failures can no longer be
merged, similar to how we have it for stateful test failures.

> We plan to merge this in the next few days and we'll send out another
notification when it's done.

## Fixing `parallel.ts`

There were two problems with the
`x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts`
script we use for running Cypress tests:

- The script was broken in elastic#162673
(here on [this
line](https://github.com/elastic/kibana/pull/162673/files#diff-9f40ced6d29c4fc2709af881680400293d8ce1bc9ebb07b9138d6d99c83c09c9R67))
- I think it has never supported situations when all tests matching a
spec pattern (such as
`./cypress/e2e/!(investigations|explore)/**/*.cy.ts`) end up being
skipped via Cypress tags (such as `@skipInServerless`)

Both the issues are fixed in this PR.

Code owners are added for this script in the CODEOWNERS file to prevent
breaking this script in future PRs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@banderror banderror

Labels
8.11 candidate Team:Detection Engine Security Solution Detection Engine Area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Security Solution] Skip all Detection & Response Cypress tests in Serverless banderror/kibana
2 participants
@maximpn @banderror