-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test examples in CI #3015
Test examples in CI #3015
Conversation
TODO: - run all the tests, not just the ones which use webpack (also an issue with CI) - fix webxr test - run in CI - share WebDriver instance between tests - maybe ditch async, since we don't really need it here and it adds a bunch of dependencies and build time.
It isn't supported in Firefox yet, which is where we're running our tests.
It wouldn't have worked anyway because at least for the moment, I'm using one WebDriver session per test, and Firefox at least only allows one session to be connected. I would like to make them share a session, in which case I could add this back, but I can't right now because Firefox hasn't implemented `LogEntry.source` yet, which is needed to figure out which log entries should fail which tests.
Although we can't test them, we can still build them.
This is needed for the `webxr` example.
This seems excessive but 10s is still sometimes failing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds reasonble to me, thanks for this!
.github/workflows/main.yml
Outdated
@@ -112,7 +112,8 @@ jobs: | |||
- uses: actions/setup-node@v2 | |||
with: | |||
node-version: '16' | |||
- run: cargo test | |||
# Run everything except the example tests; we run them later in `test_examples`. | |||
- run: RUSTFLAGS="--cfg=disable_example_tests" cargo test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given how heavyweight these tests are in system dependencies could this be an opt-in feature rather than an opt-out feature? (e.g. via required-features = ['...']
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've moved it to a separate example-tests
crate instead to avoid adding a debug-only feature to the root wasm-bindgen
crate.
This PR adds some testing of examples to CI. It doesn't actually test any of the functionality of the examples, it just runs them and checks that no errors get logged to the console. Hopefully, this should stop issues like #2963 from happening again.
I've also made CI build a couple extra examples that it didn't build before - previously only the examples that used webpack were built, but I've now made it build the rest as well by unifying on a
build.sh
for everything other than webpack.Since WebDriver doesn't support getting at logs yet, I've implemented this using the experimental new WebDriver BiDi protocol which does. Currently I've only implemented support for testing with Firefox which has it built in, but there's also a Chromium implementation if you think it's important to support that too.