Add UI test/canary that exercises "typical flow"

[RomainMuller]

Following discovery of the bug fixed in #630, it became clear we need automated UI testing around the whole user flow/journey, so that such issues don't go unnoticed in the future.

[Chriscbr]

Interesting - so the bug in #630 wouldn't have been caught by an integration test (e.g. on PRs or as a check on our internal pipeline) because the change has to actually take affect in a newly generated package before it's noticed on the UI.

I wonder if something like a Cloudwatch Synthetics broken link checker canary would make sense here, since it could potentially catch any broken links on the website in the future (and could be a lot simpler to set up)?

[Chriscbr]

So even with the above in mind, the Construct Hub should still run UI tests going through typical website user flows on a periodic basis. Running tests on pull requests or as deployment steps isn't enough since these tests can also fail due to changes in our backend data (like new packages, or backend reprocessing).

We should also try to reuse the tests we have already written with Cypress in construct-hub-webapp so that they can be used as a single source of truth. (Maybe we could add a flag to enable or disable running the tests with fixture data)

Here's a couple of ways that I thought might achieve this:

1. Use Cypress Dashboard. This would make it easy to inspect test runs and inspect the tests, but I think it might provide a lot more features than we need for our use.
2. Use GitHub actions to run UI tests against constructs.dev on a routine basis (e.g. once an hour).
3. Create our own test canary that runs in AWS. It looks like this isn't practical with AWS Lambda at the moment (see Run Cypress Headless Without Binary cypress-io/cypress#1232 (comment)), but it is possible to do with AWS CodeBuild. This could be an interesting use case for a construct, perhaps?

Note: options 1 and 2 don't provide UI test canaries for private Construct Hub. But I think option 2 might be the easiest (at least as an interim solution).

[Chriscbr]

Ah I just realized (2) probably is no good since the UI tests and our actual deployment of the website will be tied to different lifecycles. For example, we might update the integration tests along with a code change to the website, and then the tests will start failing until our actual website deployment change is live. We would also have less options for tracking and monitoring test metrics so... I guess (3) is the way to go.

[RomainMuller]

What about CloudWatch Synthetics? This is supposedly what it's intended for...?

[Chriscbr]

This issue requires some further discussion as we decide what features we want to provide for private Construct Hub, and how high priority this is, and how important it is to continue using our Cypress-based test suite.

If we want to have a UI canary running as a construct (within Construct Hub), then I believe the best path forward would be to have the Cypress test suite available at deployment time (so that the canary can run the tests corresponding to the deployed version of Construct Hub). This requires either taking construct-hub-webapp as a peer dependency, or separating the Cypress tests in construct-hub-webapp so they can be run in isolation and then bundled in the construct-hub NPM package. OR, this could be resolved by combining the repositories together so that construct-hub already includes all of the E2E testing code.

If this doesn't need to be a construct, we could consider how can we just run the E2E testing code in our internal pipeline. And if we don't want to use Cypress, then we would have to rewrite our tests with Puppeteer to work with CloudWatch Synthetics.

For now I've submitted cdklabs/construct-hub-webapp#820 as a stop gap solution.