-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Meta] Fix i18n end-to-end test #214
Comments
This repo is set up to triage issues with Tribble, but this issue couldn't be processed. If you deliberately didn't use Tribble to report this issue, you can safely ignore this warning. If you did, something's gone wrong here. |
This should make sure that all tests are always run, even if some fail, which will allow more rapid debugging and test re-running! This partially addresses #214.
Error log:
|
Also, this is less inconvenient now that the matrix is set up to not fail-fast, meaning everything is run anyway so we can actually see exactly what failed each time. |
Noting that I haven't seen this bug since capsules were merged, so I may have fixed something at some stage accidentally...closing very tentatively for now... |
Well, that was predictable. |
For about a year now, the i18n end-to-end test has been extremely patchy, and, since GHA automatically stops other test runs if one fails, it's not uncommon for a PR or commit to get failing tests even if it's perfectly valid, just because the i18n test fails. Others have failed in the past (false negatives are an unavoidable side-effect of full headless browser end-to-end tests, I get that), but none as frequently as the i18n one (to my observation).
In terms of fixing this, I think the first step is to try to get GHA to push ahead and run all the tests, even if a few fail, which would also provide far better debugging capability, and make re-runs much speedier (we usually just re-run failing tests, and if there's only one rather than 14 others that were cancelled, that's much quicker!). Beyond that, I'll have a look in the coming days at any possible improvements that could be made to the test itself to prevent the weird failure.
TODO actual error log when I have time to extract it from GHA
The text was updated successfully, but these errors were encountered: