UI Testing
We have adopted playwright as the UI test framework for helioviewer.org.
Using playwright, we can write/record tests that test helioviewer.org as an end user. Playwright is programmed with javascript to act as an end user to interact with the page. It can then compare that the application appears how we expect it to appear by doing a diff against expected screenshots, or simply using javascript to assert that some content is visible on the page.
Playwright is very developer friendly. Here is a quick reference of how to use it.
To run tests, use
npx playwright test
This will run all the tests in the tests folder based on the playwright.config.ts configuration file.
If all tests pass, the command will exit. If any tests fail, the command will automatically open the playwright report
which lets you quickly view the failing tests.
To debug individual tests:
npx playwright test --ui
This command will open playwright's GUI. From here you can select the individual test you want to run, and the browser you want to test it in. You get a full view of the test commands being run, a video report of what the web page looks during each test step, and the console and network tab of the page.
To open the test generator, make sure your development environment is running and execute
npx playwright codegen http://localhost:8080
This command assumes you're using the local helioviewer development environment. If you're using something else, you may have to specify a different url. This will open a browser and autogenerate code for your test by recording your actions on the page.
If you need to write a test for a specific device, for example an iPhone device, you can run
npx playwright codegen --device="iPhone 12" http://localhost:8080
This will open the test generator while emulating the viewport of this device.
You have to manually copy the generated code into a test file.
Immediately after integrating playwright, some obvious issues have become clear. We have discovered some race conditions, we can see quickly how there are minor (and major) visual differences on different browsers, and we can see where there is a lack of visual feedback of actions being taken. These are good issues to know about, because we can start working on fixing them, and we can make sure we don't introduce new issues by working in the dark and manually testing new features here and there.
According to our tests, there are issues with Webkit (Safari, Epiphany) on both Mac and Linux (Windows testing is coming later). We see that the input fields are colored incorrectly, and that there may be lines between the image tiles within the viewport.
At this time, the recommendation is to use Chrome or Firefox instead of a webkit based browser if you experience these issues.