Research Cypress Alternatives for testing

[Abhishek-kumar09]

Current Behavior

We test our UI e2e with cypress today, but there are few jerks with the subscriptions and graphQL, also we use the same framework in Meshery-extensions as well.

Your suggestions to new framework would be helpful.

Frameworks to research:

• Playwright PoC (and contributing docs) for e2e testing of Meshery UI #10386
• Puppeteer with Chai
• Some other ones...

Expected Behavior

Screenshots/Logs

Environment

• Host OS: Mac Linux Windows
• Platform: Docker or Kubernetes
• Meshery Server Version: stable-v
• Meshery Client Version: stable-v

---

Contributor Guides and Handbook

• 🛠 Meshery Build & Release Strategy
• 📚 Instructions for contributing to documentation
  • Meshery documentation site and source
• 🎨 Wireframes and designs for Meshery UI in Figma
• 🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

[github-actions]

This issue has been labeled with 'design-required'. Note that prior to commencing on implementation, a design specification needs to be created and reviewed for approval. See Creating a Functional Specification to create a design spec.

---

        Be sure to join the community, if you haven't yet and please leave a ⭐ star on the project 😄

[vaibhav135]

@Abhishek-kumar09 Well I am only aware about playwright, cypress (which we are already using) and puppeteer (as of now). Let's start with playwright POC first - specifically showcasing the tests for graphql subscriptions. Then we can move with others.

[MarioArriaga92]

@vaibhav135 @Abhishek-kumar09 @leecalcote https://www.linkedin.com/advice/1/how-do-you-test-graphql-subscriptions-live-updates

[MarioArriaga92]

@vaibhav135 @Abhishek-kumar09 @leecalcote https://www.linkedin.com/advice/1/how-do-you-test-graphql-subscriptions-live-updates

Can we define a basic set of test scenarios we'd like to test? That way we can produce similar test suites from each alternative (including current framework - cypress) and evaluate accordingly.

[MarioArriaga92]

@Abhishek-kumar09 @leecalcote @vaibhav135 I did found a library called laika that extends cypress so it can easily intercept (and optionally, mock or modify) each GraphQL subscription operation subscribed to by the application under test:
https://zendesk.github.io/laika/docs/usage-in-cypress

[MarioArriaga92]

@Abhishek-kumar09 @leecalcote @vaibhav135 I did found a library called laika that extends cypress so it can easily intercept (and optionally, mock or modify) each GraphQL subscription operation subscribed to by the application under test: https://zendesk.github.io/laika/docs/usage-in-cypress

Its an OSS tool by Zendesk: https://github.com/zendesk/laika
mocking is optional but the related post explains how the developer experience gets like when utilizing this tool: https://zendesk.engineering/mock-graphql-operations-with-laika-the-only-apollo-client-mocking-tool-youll-need-c116e1f62128#f6ec

[Abhishek-kumar09]

Thanks for the research items @MarioArriaga92, looking at it.

[vaibhav135]

@MarioArriaga92 Thanks for the useful links.

Btw I would like to mention that although I myself don't have much experience with cypress. But it seems that It has some problem when it comes to running in ci/cd pipeline.
One example that I found is this issue that is opened now for nearly two years and a lot of people still are facing the issue (including me few days back). Although I was able to solve it in my case in this PR. But not so for other people check here. Although I know cypress has a big community but this still makes me question the reliability of it for our usage.
With playwright although it's still new, but you can be assured that a big entity (Microsoft in this case) is backing it up. And you might not face these little issue (just my speculation).
At the end I would just say that both are good and we can achieve the same result (maybe) with both of them. So it's just boils down to the ease of use, robustness and DX. I guess maybe we should write more tests in cypress to find out if it really fulfills our needs.

Btw @leecalcote and @Abhishek-kumar09 I would like to know about your experience with cypress.

[MarioArriaga92]

@MarioArriaga92 Thanks for the useful links.

Btw I would like to mention that although I myself don't have much experience with cypress. But it seems that It has some problem when it comes to running in ci/cd pipeline. One example that I found is this issue that is opened now for nearly two years and a lot of people still are facing the issue (including me few days back). Although I was able to solve it in my case in this PR. But not so for other people check here. Although I know cypress has a big community but this still makes me question the reliability of it for our usage. With playwright although it's still new, but you can be assured that a big entity (Microsoft in this case) is backing it up. And you might not face these little issue (just my speculation). At the end I would just say that both are good and we can achieve the same result (maybe) with both of them. So it's just boils down to the ease of use, robustness and DX. I guess maybe we should write more tests in cypress to find out if it really fulfills our needs.

Btw @leecalcote and @Abhishek-kumar09 I would like to know about your experience with cypress.

@vaibhav135 good points, on the issue you mention ("Error: ESOCKETTIMEDOUT" during cy.visit command) is something very generic that can happen in different circumstances where the application under test built with an apparent "success" but is somehow unable to serve the page like in this case where an angular app was served with the "watch" flag causing the app to be re-built during test initialization and page visit timed out.

[MarioArriaga92]

@vaibhav135 #7740 is a great example, pay close attention to this comment where a common solution to most issues that result in an ESOCKETTIMEDOUT error (based on cypress-io/cypress issue 7062 reply by jennifer-shehane, responseTimeout can be increased as if page takes longer than the 30 second default timeout to load, like when next.js builds a page the first time it is visited as in Meshery UI, such error would be thrown)

[MarioArriaga92]

@vaibhav135 #7740 is a great example, pay close attention to this comment where a common solution to most issues that result in an ESOCKETTIMEDOUT error (based on cypress-io/cypress issue 7062 reply by jennifer-shehane, responseTimeout can be increased as if page takes longer than the 30 second default timeout to load, like when next.js builds a page the first time it is visited as in Meshery UI, such error would be thrown)

Why does this kind of problem happen mostly in CI environments? There could be a bug in cypress itself with certain setups and conditions but based on experience working with Cypress and Github Workflows in Meshery its mostly due to any of the following circumstances:

• problems that are not noticed in a prod/dev build during build time until the app is started and an end user would attempt to visit - either resulting in a page taking longer than expected to load or not load at all.
• Having our app test runs be against dev builds not optimized or with suboptimal configurations for their Server-Side Rendering features (Like Next.js's SSR where page HTML is generated on each request)

Is a 30 second load time acceptable? I'd say no but its not a measurement in fair conditions as CI machines usually have low amount of resources, have commodity hardware and/or could have shared resources across multiple runs/agent processes) so a separate setup and/or tool(s) can/should be used to evaluate performance. There are combinations like Cypress + Lighthouse or Cypress + browser's native Performance API to benchmark apps from an end-to-end / user perspective but that'd be a separate topic hehe.

[github-actions]

Checking in... it has been awhile since we've heard from you on this issue. Are you still working on it? Please let us know and please don't hesitate to contact a MeshMate or any other community member for assistance.

---

        Be sure to join the community, if you haven't yet and please leave a ⭐ star on the project 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity. However, you may choose to reopen this issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity. However, you may choose to reopen this issue.