Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd intermittent failure reporting to filter-intermittents #16946
Conversation
highfive
commented
May 19, 2017
|
Heads up! This PR modifies the following files:
|
|
r? @Manishearth |
| data = { | ||
| 'test_file': intermittent['test'], | ||
| 'platform': 'TODO', # TODO: need saltfs changes | ||
| 'builder': 'TODO', # TODO: need saltfs changes |
This comment has been minimized.
This comment has been minimized.
jdm
May 19, 2017
Author
Member
@aneeshusa What would be the best way to obtain the name of the builder here?
|
r=me |
| data = { | ||
| 'test_file': intermittent['test'], | ||
| 'platform': platform.system(), | ||
| 'builder': 'TODO', # TODO: need saltfs changes |
This comment has been minimized.
This comment has been minimized.
aneeshusa
May 19, 2017
Member
Do you mean a Buildbot builder (e.g. android) or a build machine (servo-linux-cross3)?
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
aneeshusa
May 19, 2017
Member
I believe we'd want to expose the Buildbot slavename property as an environment variable by using interpolation, as mentioned in servo/saltfs#597.
| (last_merge, _) = proc.communicate() | ||
|
|
||
| # Extract the issue reference from "abcdef Auto merge of #NNN" | ||
| pull_request = last_merge.split(' ')[4][1:] |
This comment has been minimized.
This comment has been minimized.
aneeshusa
May 19, 2017
Member
A bunch of this looks like it could be pulled out of the for intermittent in intermittents loop; I presume you'd want to add an early return if there are no intermittents as well.
| @@ -556,6 +566,37 @@ def filter_intermittents(self, summary, log_filteredsummary, log_intermittents, | |||
| else: | |||
| intermittents += [failure] | |||
|
|
|||
| if reporter_api: | |||
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
jdm
May 19, 2017
Author
Member
No. There's no point until I have an actual URL to give, which requires setting up the service first.
|
|
|
|
@bors-servo: treeclosed=100 |
|
@bors-servo: treeclosed- |
|
This needs servo/saltfs#739 to be merged and deployed before the API endpoint is available. |
|
r? @metajack |
|
Nevermind, Manish gave this r+ long ago. |
|
@bors-servo: r=Manishearth |
|
|
Add intermittent failure reporting to filter-intermittents We need to address servo/saltfs#671 before this can be enabled by default on the builders, but I got the pieces working locally. This collects relevant information about each recorded intermittent failure encountered during a test run and posts them to a webapp which stores them in a database for later investigation. This will allow us to track over time which intermittent test failures are the most frequent, as well as whether they only happen on a particular builder or operating system. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16946) <!-- Reviewable:end -->
|
|
jdm commentedMay 19, 2017
•
edited by larsbergstrom
We need to address servo/saltfs#671 before this can be enabled by default on the builders, but I got the pieces working locally. This collects relevant information about each recorded intermittent failure encountered during a test run and posts them to a webapp which stores them in a database for later investigation. This will allow us to track over time which intermittent test failures are the most frequent, as well as whether they only happen on a particular builder or operating system.
This change is