Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generalize the 2020 regression report to show local unexpected failures #26437

Merged
merged 2 commits into from May 7, 2020

Conversation

@SimonSapin
Copy link
Member

SimonSapin commented May 5, 2020

Example usage:

./mach test-wpt --release --layout-2020 --log-raw /tmp/servo.log
./tests/wpt/reftests-report/gen.py /tmp/servo.log
firefox ./tests/wpt/reftests-report/report.html

Produces a report similar https://community-tc.services.mozilla.com/api/index/v1/task/project.servo.layout-2020-regressions-report/artifacts/public/regressions.html, but showing unexpected reftest failures. The CI-generated one shows Layout 2020 failures (expected or not) that succeed in Layout 2013.

@highfive
Copy link

highfive commented May 5, 2020

Heads up! This PR modifies the following files:

  • @jgraham: tests/wpt/reftests-report/gen.py, tests/wpt/reftests-report/prism.js, tests/wpt/reftests-report/prism.css
@SimonSapin
Copy link
Member Author

SimonSapin commented May 5, 2020

r? @jdm

@highfive highfive assigned jdm and unassigned Manishearth May 5, 2020
@jdm
Copy link
Member

jdm commented May 5, 2020

@bors-servo
Copy link
Contributor

bors-servo commented May 5, 2020

📌 Commit 10923ca has been approved by jdm

@bors-servo
Copy link
Contributor

bors-servo commented May 5, 2020

Testing commit 10923ca with merge 74454ae...

bors-servo added a commit that referenced this pull request May 5, 2020
Generalize the 2020 regression report to show local unexpected failures

Example usage:

```
./mach test-wpt --release --layout-2020 --log-raw /tmp/servo.log
./tests/wpt/reftests-report/gen.py /tmp/servo.log
firefox ./tests/wpt/reftests-report/report.html
```

Produces a report similar https://community-tc.services.mozilla.com/api/index/v1/task/project.servo.layout-2020-regressions-report/artifacts/public/regressions.html, but showing unexpected reftest failures. The CI-generated one shows Layout 2020 failures (expected or not) that succeed in Layout 2013.
@bors-servo
Copy link
Contributor

bors-servo commented May 6, 2020

💔 Test failed - status-taskcluster

@jdm
Copy link
Member

jdm commented May 6, 2020

# Push to master:
Found task <from index> indexed at project.servo.<MagicMock name='mock.check_output().decode().strip().__radd__()' id='140578698698592'>
Found task <from index> indexed at project.servo.docker-image.263b6783749c63391a9ca609cff096214027bfb21b16c61e851197f068cd8def
Scheduled b'<new id>': Linux x64: Upload docs to GitHub Pages
Traceback (most recent call last):
  File "./etc/taskcluster/mock.py", line 61, in <module>
    decision_task.main("github-push")
  File "/repo/etc/taskcluster/decision_task.py", line 15, in main
    tasks(task_for)
  File "/repo/etc/taskcluster/decision_task.py", line 71, in tasks
    function()
  File "/repo/etc/taskcluster/decision_task.py", line 268, in layout_2020_regressions_report
    .with_index_at("layout-2020-regressions-report")
AttributeError: 'NoneType' object has no attribute 'create'
Name                               Stmts   Miss  Cover   Missing
----------------------------------------------------------------
etc/taskcluster/decision_task.py     184     36    80%   73-98, 149, 287-293, 347, 394, 398, 414, 443, 462-463, 690-697, 801
etc/taskcluster/decisionlib.py       318      6    98%   77-78, 471-472, 485-486
etc/taskcluster/mock.py               38      7    82%   63-72
----------------------------------------------------------------
TOTAL                                540     49    91%
[taskcluster 2020-05-06 00:15:14.434Z] === Task Finished ===
@@ -157,6 +157,9 @@ def __init__(self, name):

with_extra = chaining(update_attr, "extra")

def with_index_at(self, index_path):
self.routes.append("index.%s.%s" % (CONFIG.index_prefix, index_path))

This comment has been minimized.

Copy link
@jdm

jdm May 6, 2020

Member

This needs to return self.

@SimonSapin SimonSapin force-pushed the reftest-report branch from 10923ca to f524363 May 7, 2020
@SimonSapin
Copy link
Member Author

SimonSapin commented May 7, 2020

Indeed, that fixes running mock.py. Thanks!

@bors-servo r=jdm

@bors-servo
Copy link
Contributor

bors-servo commented May 7, 2020

📌 Commit f524363 has been approved by jdm

@bors-servo
Copy link
Contributor

bors-servo commented May 7, 2020

Testing commit f524363 with merge 856f03a...

@bors-servo
Copy link
Contributor

bors-servo commented May 7, 2020

☀️ Test successful - status-taskcluster
Approved by: jdm
Pushing 856f03a to master...

@bors-servo bors-servo merged commit 856f03a into master May 7, 2020
2 checks passed
2 checks passed
Community-TC (pull_request) TaskGroup: success
Details
homu Test successful
Details
@bors-servo bors-servo deleted the reftest-report branch May 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.