Unify challenge testing for Curriculum in Browser & Tooling

[raisedadead]

We do not have an overlap between how we run challenge tests (the tests that a user needs to pass when completing a challenge) in the browser and in the tooling (npm run test:curriculum).

While they should work in an ideal world. Sometimes they do not lead us to figure out workarounds like changing user stories which is not a good idea.

Older thread: #4201

[naomi-lgbt]

What's the implementation plan here - running the challenges in Cypress would bog down our CI.

[raisedadead]

The idea is to run the test cases in the browser economically. We will also need some form of the release plan. We can run one Cypress suite twice weekly (say on Sunday or Wednesday), and the release can happen the next day (Monday and Thursday).

The person doing the release can verify if some tests failed. I am sure with the flakiness of testing in general, some will. But all they have to do is manually spot-check some of the failing ones in their local setup, and we should be golden.

All this aims to improve our confidence in pushing changes out.

[ShaunSHamilton]

I am sure with the flakiness of testing in general, some will.

Is there a way to slow Cypress down?

If we only ran this once/twice per week, then we could get away with Cypress taking 3+ hours to complete, and this should reduce flakiness.

[raisedadead]

Is there a way to slow Cypress down?

You mean to run it less frequently. We could do something like a nightly build or something. But then we need to make sure we do not do more than one release at any cost in a day.

[ShaunSHamilton]

You mean to run it less frequently

No. I mean the flakiness often comes from a page not having loaded before elements on it are accessed (timeouts). So, if we just slowed down the speed with which Cypress clicks stuff, then we might have a smoother experience more apt to an actual Camper.

[ojeytonwilliams]

Yeah, that's all configurable. You can customise the various waits and the typing speed.