Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Meta] Fix i18n end-to-end test #214

Closed
arctic-hen7 opened this issue Oct 27, 2022 · 5 comments
Closed

[Meta] Fix i18n end-to-end test #214

arctic-hen7 opened this issue Oct 27, 2022 · 5 comments
Labels
author-willing-to-impl The author of this issue is willing to try to implement the solution themselves. C-bug Category: bug P-medium Priority: medium S-in-design Status: in design

Comments

@arctic-hen7
Copy link
Member

For about a year now, the i18n end-to-end test has been extremely patchy, and, since GHA automatically stops other test runs if one fails, it's not uncommon for a PR or commit to get failing tests even if it's perfectly valid, just because the i18n test fails. Others have failed in the past (false negatives are an unavoidable side-effect of full headless browser end-to-end tests, I get that), but none as frequently as the i18n one (to my observation).

In terms of fixing this, I think the first step is to try to get GHA to push ahead and run all the tests, even if a few fail, which would also provide far better debugging capability, and make re-runs much speedier (we usually just re-run failing tests, and if there's only one rather than 14 others that were cancelled, that's much quicker!). Beyond that, I'll have a look in the coming days at any possible improvements that could be made to the test itself to prevent the weird failure.

TODO actual error log when I have time to extract it from GHA

@github-actions
Copy link

This repo is set up to triage issues with Tribble, but this issue couldn't be processed. If you deliberately didn't use Tribble to report this issue, you can safely ignore this warning. If you did, something's gone wrong here.

@arctic-hen7 arctic-hen7 added C-bug Category: bug P-medium Priority: medium S-in-design Status: in design author-willing-to-impl The author of this issue is willing to try to implement the solution themselves. labels Oct 27, 2022
arctic-hen7 added a commit that referenced this issue Oct 29, 2022
This should make sure that all tests are always run, even if some fail,
which will allow more rapid debugging and test re-running!

This partially addresses #214.
@arctic-hen7
Copy link
Member Author

Error log:

---- main stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `""`,
 right: `"Hello, \u{2068}User\u{2069}!"`', examples/core/i18n/tests/main.rs:16:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@arctic-hen7
Copy link
Member Author

Also, this is less inconvenient now that the matrix is set up to not fail-fast, meaning everything is run anyway so we can actually see exactly what failed each time.

@arctic-hen7
Copy link
Member Author

Noting that I haven't seen this bug since capsules were merged, so I may have fixed something at some stage accidentally...closing very tentatively for now...

@arctic-hen7
Copy link
Member Author

Well, that was predictable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author-willing-to-impl The author of this issue is willing to try to implement the solution themselves. C-bug Category: bug P-medium Priority: medium S-in-design Status: in design
Projects
None yet
Development

No branches or pull requests

1 participant