Skip to content

/ servo

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve diagnostics for WPT failures #24981

Merged
merged 5 commits into from Dec 3, 2019
Merged

Improve diagnostics for WPT failures #24981

merged 5 commits into from Dec 3, 2019

Conversation

@SimonSapin
Copy link
Member

SimonSapin commented Dec 1, 2019 •
edited

  • Include the full output (including stdout/stderr) in the intermittent-filtered log

  • Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)

  • Fail with a specific message when a reftest screenshot is entirely white (This caused over a hundred unexpected results. A few of them in reftests that use about:blank as a reference.)

  • For failing reftests, add a message if the whole screenshot is a solid color, to help recognize instances of #24726

      ▶ FAIL [expected PASS] /css/CSS2/box-display/root-box-003.xht
  │   → /css/CSS2/box-display/root-box-003.xht 54a9df64f1476dd12020019d7cf22ac34d727bc0
  │   → /css/CSS2/box-display/root-box-003-ref.xht 636eb693bc214b6e1c64e6566c48e69e6777b946
  └   → Screenshot is solid color 0xFFFFFF for /css/CSS2/box-display/root-box-003.xht

    (The last line is new.)
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 1, 2019

@bors-servo try=wpt
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

⌛ Trying commit 308e4f7 with merge c831b6f...
bors-servo added a commit that referenced this pull request Dec 1, 2019
@bors-servo
Auto merge of #24981 - servo:wpt-unexpected, r=<try>
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
c831b6f 
Improve diagnostics for WPT failures

* Include the full output (including stdout/stderr) in the intermittent-filtered log
* Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)
* Fail with a specific message when a reftest screenshot is entirely white, to help recognize instances of #24726
@servo-wpt-sync servo-wpt-sync mentioned this pull request Dec 1, 2019
Merged
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 1, 2019

Opened new PR for upstreamable changes.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

💔 Test failed - status-taskcluster
@SimonSapin SimonSapin force-pushed the wpt-unexpected branch from 308e4f7 to 1e1f191 Dec 1, 2019
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 1, 2019

@bors-servo try=wpt
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

⌛ Trying commit 1e1f191 with merge dd7003b...
bors-servo added a commit that referenced this pull request Dec 1, 2019
@bors-servo
Auto merge of #24981 - servo:wpt-unexpected, r=<try>
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
dd7003b 
Improve diagnostics for WPT failures

* Include the full output (including stdout/stderr) in the intermittent-filtered log
* Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)
* Fail with a specific message when a reftest screenshot is entirely white, to help recognize instances of #24726
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 1, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

💔 Test failed - status-taskcluster
@SimonSapin SimonSapin force-pushed the wpt-unexpected branch from 1e1f191 to 10a3572 Dec 1, 2019
@highfive highfive removed the S-tests-failed label Dec 1, 2019
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 1, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 1, 2019

@bors-servo try=wpt
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

⌛ Trying commit 10a3572 with merge e012478...
bors-servo added a commit that referenced this pull request Dec 1, 2019
@bors-servo
Auto merge of #24981 - servo:wpt-unexpected, r=<try>
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
e012478 
Improve diagnostics for WPT failures

* Include the full output (including stdout/stderr) in the intermittent-filtered log
* Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)
* Fail with a specific message when a reftest screenshot is entirely white, to help recognize instances of #24726
@bors-servo
Copy link
Contributor

bors-servo commented Dec 1, 2019

💔 Test failed - status-taskcluster
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 1, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 1, 2019

Perfect, the two filtered unexpected results demonstrate the changes in this PR:

This is ready for review.

r? @jdm
CC @jgraham for changes in tests/wpt/web-platform-tests/tools/wptrunner
@SimonSapin SimonSapin force-pushed the wpt-unexpected branch from 1a418bc to d0785e0 Dec 2, 2019
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 2, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@SimonSapin
Improve formatting of multi-line "message" in Servo’s WPT log formatter
40ce8c7
@SimonSapin SimonSapin force-pushed the wpt-unexpected branch from d0785e0 to 46b122f Dec 3, 2019
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 3, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@jdm
Copy link
Member

jdm commented Dec 3, 2019

@bors-servo try=wpt
@bors-servo
Copy link
Contributor

bors-servo commented Dec 3, 2019

⌛ Trying commit 46b122f with merge 955431f...
bors-servo added a commit that referenced this pull request Dec 3, 2019
@bors-servo
Auto merge of #24981 - servo:wpt-unexpected, r=<try>
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
955431f 
Improve diagnostics for WPT failures

* Include the full output (including stdout/stderr) in the intermittent-filtered log
* Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)
* <del>Fail with a specific message when a reftest screenshot is entirely white</del> (This caused over a hundred unexpected results. A few of them in reftests that use `about:blank` as a reference.)
* For failing reftests, add a message if the whole screenshot is a solid color, to help recognize instances of #24726

  ```
    ▶ FAIL [expected PASS] /css/CSS2/box-display/root-box-003.xht
    │   → /css/CSS2/box-display/root-box-003.xht 54a9df64f1476dd12020019d7cf22ac34d727bc0
    │   → /css/CSS2/box-display/root-box-003-ref.xht 636eb693bc214b6e1c64e6566c48e69e6777b946
    └   → Screenshot is solid color 0xFFFFFF for /css/CSS2/box-display/root-box-003.xht
  ```
  (The last line is new.)
@bors-servo
Copy link
Contributor

bors-servo commented Dec 3, 2019

💔 Test failed - status-taskcluster
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 3, 2019

I think what happened in #24981 (comment) was that I forced-push from the wrong machine an earlier version of this branch. I’ve canceled this last try=wpt run since it only tests Linux which I’d already tested on the more recent version of the branch, and there seems to also be some Python errors on macOS. I’m testing on the try-taskcluster branch at the moment.
SimonSapin added 2 commits Dec 1, 2019
@SimonSapin
wptrunner: detect solid color screenshots for failing reftests
8dc703f 
The output looks like this:

```
  ▶ FAIL [expected PASS] /css/CSS2/box-display/root-box-003.xht
  │   → /css/CSS2/box-display/root-box-003.xht 54a9df64f1476dd12020019d7cf22ac34d727bc0
  │   → /css/CSS2/box-display/root-box-003-ref.xht 636eb693bc214b6e1c64e6566c48e69e6777b946
  └   → Screenshot is solid color 0xFFFFFF for /css/CSS2/box-display/root-box-003.xht
```
@SimonSapin
Add a test counter in WPT’s non-interactive output
Loading status checks…
f0b970d
@SimonSapin SimonSapin force-pushed the wpt-unexpected branch from 46b122f to f0b970d Dec 3, 2019
@highfive highfive removed the S-tests-failed label Dec 3, 2019
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 3, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@SimonSapin
Fix Python Unicode error on macOS CI
Loading status checks…
14f049d
@servo-wpt-sync
Copy link
Collaborator

servo-wpt-sync commented Dec 3, 2019

Transplanted upstreamable changes to existing PR.

Completed upstream sync of web-platform-test changes at web-platform-tests/wpt#20549.
@SimonSapin
Copy link
Member Author

SimonSapin commented Dec 3, 2019

Alright, I think this should be good now. r? @jdm, since you might have reviewed the obsolete branch
@jdm
Copy link
Member

jdm commented Dec 3, 2019

@bors-servo r+
@bors-servo
Copy link
Contributor

bors-servo commented Dec 3, 2019

📌 Commit 14f049d has been approved by jdm
@bors-servo
Copy link
Contributor

bors-servo commented Dec 3, 2019

⌛ Testing commit 14f049d with merge d99850f...
bors-servo added a commit that referenced this pull request Dec 3, 2019
@bors-servo
Auto merge of #24981 - servo:wpt-unexpected, r=jdm
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
d99850f 
Improve diagnostics for WPT failures

* Include the full output (including stdout/stderr) in the intermittent-filtered log
* Print the intermittent-filtered log at the end of the main log (which is one less click to reach from Taskcluster’s task view, compared to other task artifacts)
* <del>Fail with a specific message when a reftest screenshot is entirely white</del> (This caused over a hundred unexpected results. A few of them in reftests that use `about:blank` as a reference.)
* For failing reftests, add a message if the whole screenshot is a solid color, to help recognize instances of #24726

  ```
    ▶ FAIL [expected PASS] /css/CSS2/box-display/root-box-003.xht
    │   → /css/CSS2/box-display/root-box-003.xht 54a9df64f1476dd12020019d7cf22ac34d727bc0
    │   → /css/CSS2/box-display/root-box-003-ref.xht 636eb693bc214b6e1c64e6566c48e69e6777b946
    └   → Screenshot is solid color 0xFFFFFF for /css/CSS2/box-display/root-box-003.xht
  ```
  (The last line is new.)
@bors-servo
Copy link
Contributor

bors-servo commented Dec 3, 2019

☀️ Test successful - status-taskcluster
Approved by: jdm
Pushing d99850f to master...
@bors-servo bors-servo merged commit 14f049d into master Dec 3, 2019
2 checks passed
2 checks passed
Community-TC (pull_request) TaskGroup: success
Details
homu Test successful
Details
@bors-servo bors-servo deleted the wpt-unexpected branch Dec 3, 2019
@jdm jdm mentioned this pull request Dec 3, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jdm jdm

Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
@SimonSapin @bors-servo @servo-wpt-sync @jdm @highfive
You can’t perform that action at this time.