Research - Cypress for End-to-End Testing

[jamescd18]

Research Cypress for potentially using them for end-to-end (e2e) testing. Potentially consider alternative packages like CodeceptJS to achieve the same goals. Consider compatibility with React or Next.js.

Put longer notes here in this issue, along with potential sample code and any other thoughts. And then you can put a couple bullets summary / thoughts in this document.

See Northeastern-Electric-Racing/NER-PM-Dashboard#114 as an excellent and thorough example.

[eshwaribhide]

After looking into things a bit, I think Cypress should be used over CodeceptJS. I think either is fine, but Cypress is definitely a lot more popular and well-documented and is supposed to be "easy" to use, so I'll just focus on looking into Cypress.

Here is an article comparing Cypress and CodeceptJS.

[eshwaribhide]

Here is a Github Action for running Cypress tests. Another article describing it. You can install Cypress locally through a single npm command.

Cypress seems to market itself as "easy" to use. The "language of Cypress tests" is kind of just Javascript; also it has a "Time Travel" feature, meaning when your tests run it takes "snapshots" so you can view different states your application went through while testing (here is more information). Another feature is the ability to take manual screenshots, through this command
cy.screenshot().
You can also debug your tests from DevTools. Also, Cypress can be run in many different browsers...but not Safari.

Here is some documentation about simulating user interactions. I think in general pretty much everything you need to learn Cypress is on their website; there is a lot of documentation. Cypress is indeed compatible with React, through this. Previously it was a repo by itself called cypress-react-unit-test but apparently it moved. It's interesting because on the official Cypress website they have an example of configuring Cypress for React but seem to reference the old repo.

[jamescd18]

This is awesome to hear! My follow-up questions: what's the pricing and limitations of Cypress? What is the web UI needed for? What kind of setup do we need to do in addition to a GitHub Action.

[eshwaribhide]

In terms of the web UI, I'm not sure if you're referring to the Cypress Test Runner UI or if you're referring to the Cypress Dashboard. The Cypress Test Runner is just a basic UI where you can basically see all the different tests you have and then just click on test names in order to launch / run them in a browser.

The Cypress Dashboard is something different, I guess. You don't really need the Cypress Dashboard, but they heavily advertise it (I guess because there are paid options). It helps give you a visualization of your test results and provides some insights into the results. From their website, "With Cypress Dashboard you’ll find video recordings, screenshots, logs, test case history, test analytics, flaky test detection, and more." I think the most important part is video recordings. Also, it comes with GitHub integration ("reports status of test runs as GitHub commit status checks for every run group or every spec file in your test-suite") and Slack Integration (sends you real-time updates of test results to Slack).

In terms of pricing, the Cypress Test Runner is completely free and the Cypress Dashboard has a free version. It allows you <=500 test recordings each month and supports up to 3 users, where "A user is anyone with a login to the Cypress Dashboard that has been granted permission to see test results for your organization." If you want to get a paid plan, the next tier is $99 a month, which is a lot, and literally all it gives you is support for up to 5 users (so just 2 more lol) but then gives you 50x as many test recordings at <=25000 test recordings per month (which sounds like a loooooooot). Also, if you give them $25 more in addition to the $99 a month, they'll give you an SSO add-on option, which I don't think we need. See this for more info.

[eshwaribhide]

In terms of setting up Cypress, its information should be added to package.json (just run npm install cypress --save-dev I guess) and then you can open the Cypress Test Runner through npx cypress open. This will open a window (the Cypress Test Runner). In your project, there should be a cypress directory which will have an integration subdirectory, which is where your tests should live. Whenever you create a test file and save it, then it should show up on the Cypress Test Runner screen, and when you click its name it executes the test and shows you the results. I feel like the Cypress Test Runner kind of reminds me of Selenium (I've never really used it but kind of have a vague idea of it).

[eshwaribhide]

In terms of the Dashboard, here is an example of what it looks like...what kind of things you can get with it. The Dashboard allows you to see video recordings of your test, along with other things. There are steps you need to take in order to set up your project to record. To view these recordings and other insights, then you'll have to login to the Dashboard (once you create an account).

From the website, "You typically want to record when running tests in Continuous Integration, but you can also record your tests when running locally." I believe I linked a GitHub action for Cypress tests above; it looks like it is possible to do this recording with GitHub actions as CI.

Honestly, the Dashboard seems cool; I'm just wondering if it is more than we need and whether the Test Runner is enough. What do you think?

[jamescd18]

Wow, that's a ton of great info! Is it possible to get started with cypress + GitHub actions CI without the Dashboard account? If the CI works without the account, and can be added later, then I think we should just go ahead and set things up and add the account Dashboard later. Recording tests really won't be a biggie until we actually have something worthwhile to test.

[eshwaribhide]

I don't think you need the Dashboard account in order to get started with Cypress Test Runner + GitHub Actions.