Skip to content

Commit

Permalink
Build: Update browserstack-runner to fix QUnit.todo support
Browse files Browse the repository at this point in the history
It is hard to believe, but we actually never had a `QUnit.todo()`
test as part of the main index.html test suite that we run in browsers.

There were secondary tests that only run in Headless Chromium via
grunt-contrib-qunit, and CLI tests, but none that we run cross browser.

In commit 47576c1 I added test/main/each.js to index.html, not
realizing it was the first such case, and this broke the CI build for
all browsers.

* js-reporters 2.1 adds support for QUnit.todo in the adapter
  so that it correctly emits the `runEnd` event to count these toward
  `testCounts.todo` instead of toward `testCounts.failed`.
  js-reporters/js-reporters#140.

* browserstack-runner 0.9.5-qunitjs.1 updates its copy of js-reporters
  to this version, and also updates its error formatter to not print
  details for error objects from non-failed tests.

Fixes #1622.
  • Loading branch information
Krinkle committed Jun 7, 2021
1 parent b794248 commit 6cf92c9
Show file tree
Hide file tree
Showing 5 changed files with 39 additions and 12 deletions.
7 changes: 7 additions & 0 deletions .github/workflows/browsers-quick.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ jobs:
npm ci
npm run build
# To debug locally:
#
# ```
# $ export BROWSERSTACK_USERNAME="***"
# $ export BROWSERSTACK_KEY="***"
# $ BROWSERSTACK_JSON=build/browserstack-debug.json npm run browserstack
# ```
- name: Tests
run: npm run browserstack
env:
Expand Down
11 changes: 11 additions & 0 deletions build/browserstack-debug.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"test_framework": "qunit",
"test_path": ["test/index.html"],
"test_server_port": "8899",
"exit_with_fail": true,
"timeout": 600,
"browsers": [
"firefox_current",
"chrome_current"
]
}
20 changes: 10 additions & 10 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@
"@rollup/plugin-commonjs": "^18.0.0",
"@rollup/plugin-node-resolve": "^11.2.1",
"@rollup/plugin-replace": "^2.4.2",
"@qunitjs/browserstack-runner": "0.9.4-qunitjs.2",
"@qunitjs/browserstack-runner": "0.9.5-qunitjs.1",
"coveralls": "^3.1.0",
"eslint": "7.23.0",
"eslint-config-jquery": "^3.0.0",
Expand Down
11 changes: 10 additions & 1 deletion src/test.js
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,13 @@ Test.prototype = {
module.stats.all += this.assertions.length;

for ( let i = 0; i < this.assertions.length; i++ ) {

// A failing assertion will counts toward the HTML Reporter's
// "X assertions, Y failed" line even if it was inside a todo.
// Inverting this would be similarly confusing since all but the last
// passing assertion inside a todo test should be considered as good.
// These stats don't decide the outcome of anything, so counting them
// as failing seems the most intuitive.
if ( !this.assertions[ i ].result ) {
bad++;
config.stats.bad++;
Expand All @@ -316,7 +323,9 @@ Test.prototype = {
incrementTestsRun( module );
}

// Store result when possible
// Store result when possible.
// Note that this also marks todo tests as bad, thus they get hoisted,
// and always run first on refresh.
if ( storage ) {
if ( bad ) {
storage.setItem( "qunit-test-" + moduleName + "-" + testName, bad );
Expand Down

0 comments on commit 6cf92c9

Please sign in to comment.