[Proposal] E2E visual test environment

[emadurandal]

This PR proposes an E2E (Visual) Regression testing environment using existing examples.
(under discussion in #7697)

Usage

I have introduced some packages, please npm install them.

Since 'examples/cache/metadata.mjs' is referenced to extract test items, so please npm run build in advance in the examples directory.

Then, run the following command on project root.

$ npm run test:e2e 

The examples local server will then start automatically and the E2E test will begin.

On the first run, the rendered images are saved as the "correct images".
The second and subsequent runs will compare the images with the correct images created in the first run.

The following command will update the correct images.

$ npm run test:e2e:update

Current Issues

• Many existing examples are animated or have a moving camera. They are likely to fail the test due to timing issues with when the screenshot is taken.
  • It would be good to add an option to examples to disable animation for testing purposes.
• I think there are some things that are simply not suitable for testing, so we should be able to specify exclude in some way as well.

This PR is still rough around the edges, so your feedback is welcome.

I confirm I have read the contributing guidelines and signed the Contributor License Agreement.

[emadurandal]

Further updates.

Only examples with @config E2E_TEST at the beginning of ~~~~.examples.mjs are eligible for E2E testing.

I have also added an npm script to run only one test.

npm run test:e2e:partial --path=graphics/shapes

[mvaligursky]

@kpal81xd any thoughts on this?

[emadurandal]

Further updates.

• Supports running on Windows
• Switched to headless mode
• Added correct images

[kpal81xd]

Nice work!

We actually had something similar beforehand for the Editor where we compare screenshots pixel by pixel. However over time updating the testing suite and the screenshots with improvements ended up requiring more work than manually checking the examples ourselves hence we dropped it.

In addition as you have mentioned in your 'Current issues' a lot of our examples are not strictly fixed rendered perspectives they include animations such as all the particle examples (and this doesn't include take into account randomness than might occur).

One idea I had thought of was instead of storing the screenshots would be a system to compare the examples from two commits through recording them both and comparing the videos. Of course some leniency will be required due to rendering differences and some seeding for the randomness but would be a good guide for general comparisons

[emadurandal]

Thanks for the feedback. I understand the situation. It's not easy to create the best solution.

One idea I had thought of was instead of storing the screenshots would be a system to compare the examples from two commits through recording them both and comparing the videos.

I tried to find a video version of the jest-imege-snapshot, but unfortunately there does not seem to be such a version.

I leave it to you all to handle this PR. Hopefully this is the beginning of a discussion on good solutions.