Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement assert_precondition (RFC) #19844

Closed
foolip opened this issue Oct 23, 2019 · 11 comments
Closed

Implement assert_precondition (RFC) #19844

foolip opened this issue Oct 23, 2019 · 11 comments
Assignees

Comments

@foolip
Copy link
Member

@foolip foolip commented Oct 23, 2019

This issue is to implement RFC #16: 'PRECONDITION_FAILED' subtest result (status)

There are at least two parts to this:

Then there will be many tests that could be updated a few of them included in #16689.

@foolip
Copy link
Member Author

@foolip foolip commented Oct 23, 2019

The wpt side of this change should also update the documentation added in #19628.

@foolip
Copy link
Member Author

@foolip foolip commented Oct 23, 2019

The mozlog changes are up for review in https://phabricator.services.mozilla.com/D50175

@foolip
Copy link
Member Author

@foolip foolip commented Oct 24, 2019

https://bugzilla.mozilla.org/show_bug.cgi?id=1589056 is now done, and the next step is for @jgraham to release mozlog 5.0.

@foolip
Copy link
Member Author

@foolip foolip commented Oct 25, 2019

@foolip
Copy link
Member Author

@foolip foolip commented Oct 25, 2019

Oops, we're going to get these errors:

ERROR: mozrunner 7.6.0 has requirement mozlog~=4.2.0, but you'll have mozlog 5.0 which is incompatible.
ERROR: mozprofile 2.3.0 has requirement mozlog~=4.2.0, but you'll have mozlog 5.0 which is incompatible.
ERROR: mozversion 2.1.0 has requirement mozlog~=4.0, but you'll have mozlog 5.0 which is incompatible.

@jgraham how can we resolve this? Do those also need new releases to fix this? Shall I send a mozilla-central patch that bumps the minor version of each?

@foolip
Copy link
Member Author

@foolip foolip commented Oct 25, 2019

@foolip
Copy link
Member Author

@foolip foolip commented Oct 28, 2019

Those three packages now have updated releases. Thanks @jgraham!

@foolip
Copy link
Member Author

@foolip foolip commented Oct 28, 2019

mozlog itself updated in #19884

foolip added a commit that referenced this issue Oct 30, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
foolip added a commit that referenced this issue Oct 30, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
foolip added a commit that referenced this issue Oct 30, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
@foolip
Copy link
Member Author

@foolip foolip commented Nov 1, 2019

This came up at our improving interop with WPT talk at https://webkit.org/meeting/ today.

When this is resolved, if there are tests that pass spuriously in WebKit, or that fail when they shouldn't have been run, then assert_precondition could be used to allow filtering out those cases in wpt.fyi.

@cdumez @othermaciej @smfr @rniwa do you have specific cases for me to look at? I could try fixing a few of those as part of proving that this all works end-to-end.

foolip added a commit that referenced this issue Nov 2, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
@smfr
Copy link
Contributor

@smfr smfr commented Nov 2, 2019

Any test in ambient-light, battery-status, bluetooth, vibration on Safari.

@foolip
Copy link
Member Author

@foolip foolip commented Nov 4, 2019

Thanks @smfr, I've included one spurious vibration pass in #19993 to show how it works, and that it works.

foolip added a commit that referenced this issue Nov 6, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
foolip added a commit that referenced this issue Nov 6, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
foolip added a commit that referenced this issue Nov 8, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
@foolip foolip closed this in #19993 Nov 8, 2019
foolip added a commit that referenced this issue Nov 8, 2019
This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of #16689.

Fixes #19844.
moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue Nov 29, 2019
…ndition`, a=testonly

Automatic update from web-platform-tests
[testharness.js] introduce `assert_precondition`

This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of web-platform-tests/wpt#16689.

Fixes web-platform-tests/wpt#19844.

--
[vibration] use `assert_precondition` to avoid spurious pass

This demonstrates the use of `assert_precondition` for subtests.

--

wpt-commits: 17543bc30bdf65838e9ec8e3b3b6fd210dced66e, a90e89fb031b0b357305ac9d874968d8451b1181
wpt-pr: 19993
xeonchen pushed a commit to xeonchen/gecko that referenced this issue Nov 29, 2019
…ndition`, a=testonly

Automatic update from web-platform-tests
[testharness.js] introduce `assert_precondition`

This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of web-platform-tests/wpt#16689.

Fixes web-platform-tests/wpt#19844.

--
[vibration] use `assert_precondition` to avoid spurious pass

This demonstrates the use of `assert_precondition` for subtests.

--

wpt-commits: 17543bc30bdf65838e9ec8e3b3b6fd210dced66e, a90e89fb031b0b357305ac9d874968d8451b1181
wpt-pr: 19993
gecko-dev-updater pushed a commit to marco-c/gecko-dev-comments-removed that referenced this issue Nov 30, 2019
…ndition`, a=testonly

Automatic update from web-platform-tests
[testharness.js] introduce `assert_precondition`

This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of web-platform-tests/wpt#16689.

Fixes web-platform-tests/wpt#19844.

--
[vibration] use `assert_precondition` to avoid spurious pass

This demonstrates the use of `assert_precondition` for subtests.

--

wpt-commits: 17543bc30bdf65838e9ec8e3b3b6fd210dced66e, a90e89fb031b0b357305ac9d874968d8451b1181
wpt-pr: 19993

UltraBlame original commit: 9b458e91114235eb43711dbac7a3acfe30c83fce
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified-and-comments-removed that referenced this issue Nov 30, 2019
…ndition`, a=testonly

Automatic update from web-platform-tests
[testharness.js] introduce `assert_precondition`

This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of web-platform-tests/wpt#16689.

Fixes web-platform-tests/wpt#19844.

--
[vibration] use `assert_precondition` to avoid spurious pass

This demonstrates the use of `assert_precondition` for subtests.

--

wpt-commits: 17543bc30bdf65838e9ec8e3b3b6fd210dced66e, a90e89fb031b0b357305ac9d874968d8451b1181
wpt-pr: 19993

UltraBlame original commit: 9b458e91114235eb43711dbac7a3acfe30c83fce
gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified that referenced this issue Nov 30, 2019
…ndition`, a=testonly

Automatic update from web-platform-tests
[testharness.js] introduce `assert_precondition`

This depends on mozlog 5.0 for the new PRECONDITION_FAILED status:
https://bugzilla.mozilla.org/show_bug.cgi?id=1589056

Implements web-platform-tests/rfcs#16.

Includes parts of web-platform-tests/wpt#16689.

Fixes web-platform-tests/wpt#19844.

--
[vibration] use `assert_precondition` to avoid spurious pass

This demonstrates the use of `assert_precondition` for subtests.

--

wpt-commits: 17543bc30bdf65838e9ec8e3b3b6fd210dced66e, a90e89fb031b0b357305ac9d874968d8451b1181
wpt-pr: 19993

UltraBlame original commit: 9b458e91114235eb43711dbac7a3acfe30c83fce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.