[Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure

[MadameSheema]

UPDATE!

• It has been finally decided to move all the Security Solution Cypress tests to the x-pack/test/security_solution_cypress folder in order to avoid circular dependency.

AFTER MERGING THIS PR:

• All the existing and new tests needs to have a tag to be executed on the environment you expect, if not, the test is not going to be executed
• The tests marked as serverless are going to be executed on the FTR serverless environment when a PR is opened
• Teams should be extremely careful when adding new tests to make sure we are not introducing flakiness. Note that our serverless tests are initially executed on ALL the PRs, so we should not create blockers to other teams and our tests are not executed just when something is changed in our code.
• It is extremely forbidden to add conditionals on assertions or actions to make a test work in both environments. That is a red flag warning you about having to create a specific test for it.
• Specific serverless tests should be written on security_solution_cypress/cypress/e2e folder of each specific team, and should be tagged just with @serverless
• Teams are responsible to investigate the tests marked as @brokenInServerless, I can give support and advice.
• Use yarn cypress:open:ess to develop and debug tests on ESS environment
• User yarn cypress:open:serverless to develop and debug tests on FTR serverless environment

---

Summary

As mentioned on the ticket #161537, Analyst Experience Teams are going to need to start adding serverless tests.
We already have a lot of existing tests so taking into consideration the time restrictions and the head-ache of having to maintain duplicated code, we’d love to reuse the ones we can.

For Cypress specific, we explored two different ideas given by @semd and @patrykkopycinski with their support and advice (Thanks ❤️):


• To have different folders. One for serverless specific, one for ESS specific, and one for tests that can be executed in both environments.
• To use tags to know which tests can be reused.

After exploring both options, finally it was decided to implement second one, because:

• It is fast to setup
• Easy to reuse tests
• The impact on the development flow is minimal
• Taking into consideration all the above, we are not going to lose development engagement around adding ore creating new tests.

In this PR we are implementing the solution:

- We are going to maintain all the tests on the same folder to don’t disrupt developers flow, so specific Cypress serverless tests should be added to the regular folder.

• Cypress tests are placed now on x-pack/test/security_solution_cypress folder
• We are adding cypress grep (https://github.com/cypress-io/cypress/tree/develop/npm/grep) to enable tagging inside Cypress.
• We are adding two new cypress configurations for serverless-specific executions
• We are adding new scripts for serverless executions
• We are adding the new executions to buildkite so they are part of the CI process
• We are removing the waitForPageToBeLoaded method. This is because cannot be reused for both ESS and Serverless and it has been demonstrated to not be useful since Cypress waits for an element to be present before performing an action.
• We are tagging with @ess the tests that are executed on ESS environment.
• We are tagging with @serverless the tests that are executed on serverless environment.
• We are tagging with @brokenInServerless the tests that should be executed on serverless but currently are failing on that enviroment.
• We are updating the readme.
  


IMPORTANT!!!!!


Once this PR is merged:

• All the existing and new tests needs to have a tag to be executed on the environment you expect, if not, the test is not going to be executed
• The tests marked as serverless are going to be executed on the FTR serverless environment when a PR is opened
• Teams should be extremely careful when adding new tests to make sure we are not introducing flakiness. Note that our serverless tests are initially executed on ALL the PRs, so we should not create blockers to other teams and our tests are not executed just when something is changed in our code.
• It is extremely forbidden to add conditionals on assertions or actions to make a test work in both environments. That is a red flag warning you about having to create a specific test for it.
• Specific serverless tests should be written on the regular cypress folder and should be tagged just with @serverless
• Teams are responsible to investigate the tests marked as @brokenInServerless, I can give support and advice.
• Use yarn cypress:open yarn cypress:open:serverless to develop and debug tests on ESS environment
• User yarn cypress:open:serverless to develop and debug tests on FTR serverless environment

[banderror]

I couldn't run Cypress tests locally for ESS.

I'm in x-pack/test/security_solution_cypress, I run cypress:open:ess. Then ES and Kibana start apparently successfully, after that it crashes with the following error, and the test runner doesn't start:

 warn Failure loading service "security"
ERROR TimeoutError: Request timed out
          at SniffingTransport.request (/Users/georgii/Code/elastic/kibana-pr1/node_modules/@elastic/transport/src/Transport.ts:584:17)
          at processTicksAndRejections (node:internal/process/task_queues:95:5)
          at Security.putRole (/Users/georgii/Code/elastic/kibana-pr1/node_modules/@elastic/elasticsearch/src/api/api/security.ts:1391:12)
          at ensureSystemIndicesUser (system_indices_user.ts:33:3)
          at createSystemIndicesUser (system_indices_user.ts:78:3)
          at SecurityServiceProvider (security.ts:32:3)
          at provider_collection.ts:53:13
          at async Promise.all (index 9)
          at ProviderCollection.loadAll (provider_collection.ts:48:5)
          at parallel.ts:347:13
          at withProcRunner (with_proc_runner.ts:29:5)
          at concurrency (parallel.ts:181:11)
          at /Users/georgii/Code/elastic/kibana-pr1/node_modules/p-map/index.js:57:22
 info [kibana] exited with null after 156.8 seconds
ERROR UNHANDLED ERROR
ERROR Error: Failure initializing 1 service(s)
          at ProviderCollection.loadAll (provider_collection.ts:64:13)
          at processTicksAndRejections (node:internal/process/task_queues:95:5)
          at parallel.ts:347:13
          at withProcRunner (with_proc_runner.ts:29:5)
          at concurrency (parallel.ts:181:11)
          at /Users/georgii/Code/elastic/kibana-pr1/node_modules/p-map/index.js:57:22
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I also noticed that you deleted the Cypress burn CI job, what was the reason for doing this?

[banderror]

Why deleted? This job was just recently added by @patrykkopycinski to help us reveal flakiness

[MadameSheema]

As I changed all the files the execution was taking extremely long, cancel it was not helping either. The idea is to enable it again on a follow-up PR.

[banderror]

Okay, sounds good 👍

[banderror]

This is a bit suspicious dependency from a plugin to test code outside of plugins. cc @PhilippeOberti

[banderror]

The issue mentioned in #162698 (review) appeared to happen with a few folks who are on Intel-based Macs (internal slack thread). The suggested workaround is to run tests against manually started ES and Kibana instances (slack):

Hey, one workaround, it’s to run cypress tests agains you local es/kibana
So if you have es/kibana running locally, you can run cypress tests with this command

CYPRESS_CYPRESS_BASE_URL="http://elastic:changeme@localhost:5601/"  CYPRESS_BASE_URL="http://elastic:changeme@localhost:5601/" CYPRESS_ELASTICSEARCH_URL="http://system_indices_superuser:changeme@localhost:9200/" CYPRESS_ELASTICSEARCH_USERNAME="system_indices_superuser" CYPRESS_ELASTICSEARCH_PASSWORD="changeme" baseUrl="http://elastic:changeme@localhost:5601/" BASE_URL="http://elastic:changeme@localhost:5601/" ELASTICSEARCH_URL="http://system_indices_superuser:changeme@localhost:9200/" ELASTICSEARCH_USERNAME="system_indices_superuser" ELASTICSEARCH_PASSWORD="changeme" ELASTICSEARCH_URL_WITH_AUTH="http://system_indices_superuser:changeme@localhost:9200/" FLEET_SERVER_URL="http://localhost:8220/" KIBANA_URL="http://localhost:5601/" KIBANA_URL_WITH_AUTH="http://elastic:changeme@localhost:5601/" KIBANA_USERNAME="elastic" KIBANA_PASSWORD="changeme" yarn cypress open --config-file ./cypress/cypress.config.ts

Also this command don’t work for me until I commented this in kibana.dev.yaml

# server.name: kibana
# server.host: "0.0.0.0"

I'll follow up in those threads to see if we can fix this.

I also added a few fixes to the branch addressing my previously posted comments.

Thank you @MadameSheema. Go reusable Cypress tests! 🤘 🤘

[gergoabraham]

change in 24da233 looks good! 👍

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 47d9dd6

Failed CI Steps

• Threat Intelligence Cypress Tests #2
• Defend Workflows Cypress Tests #1

Test Failures

• [job] [logs] Defend Workflows Cypress Tests #1 / Endpoint Policy Response from Endpoint List page should display policy response with errors should display policy response with errors

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	466	449	-17

Total ESLint disabled count

id	before	after	diff
securitySolution	532	515	-17

History

• 💔 Build #150568 failed 24da233
• 💔 Build #150557 failed f25e362
• 💛 Build #150446 was flaky 94c9a45
• 💔 Build #150429 failed 1fbeca3
• 💔 Build #150418 failed 835a35d
• 💔 Build #150248 failed d36e8ba

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema