Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd ./mach filter-intermittents for catching intermittents on CI #14331
Conversation
highfive
commented
Nov 23, 2016
|
Heads up! This PR modifies the following files:
|
|
This needs auth because Github gives you 20 unauthenticated search API requests per minute or so, which I easily hit when testing this. Github also lets you use Oauth for the API, if you think that's easier to set up. |
Nice!
Yes, we should do that instead. Highfive can't search (at least, not until we go for Github integrations) because it lives on @jdm's token, and is already hitting the API limit.
Also, yes, it'd be nice to have a dedicated auth token for these search queries. cc'ing @aneeshusa for buildbot ideas |
|
With secret provisioning in saltfs but the steps in servo, I'd prefer to use environment variables than a file for passing authorization information. As I mentioned in #10504 (comment), I still think it's easier to get Highfive to do all the log parsing and filtering at one time. Plus, since Highfive is continuously running, we can cache information about intermittents there, watch for issue events to update its cache when filing a new intermittent or closing/fixing one, etc. That would be much more API-friendly than searching for each failure for each build. |
Yes, but then highfive has to trigger a rebuild (which means that the pain of intermittents doesn't quite go away). In this case the same build can still be accepted. We'd need to change homu otherwise. |
|
Oh, I didn't think that far, that does make sense. Feel free to change |
|
FYI, in its current incarnation highfive is not continuously running and stores no state between webhook invocations. |
|
I'm OK with this, but are you not going to change buildbot_steps.yml to include running it in this PR? |
|
I guess we need to:
|
|
@bors-servo r+ |
|
|
Add ./mach filter-intermittents for catching intermittents on CI cc @metajack The plan here is to run this on the error summary of the logs after each WPT/CSS run, as `./mach filter-intermittents wpt-errorsummary.log --output filtered-errorsummary.log --auth /path/to/authfile` The `filtered-errorsummary.log` file will be a buildbot artifact for this run, and can be used for updating test results. We change WPT/CSS runs to not cause test failures on Buildbot; instead; the test failure will be caused by this job. We should at some point add a separate highfive task which ccs the appropriate bugs and tracks consistent failures. (We can change mach filter-intermittents to output a second file of intermittents matched to bug numbers to make this easy) A small issue with this is that this simple task might mask failures of the WPT harness itself. We have a separate test that tests if the test harness works though. (This test will fail if the log files it requires don't exist, perhaps that's enough to solve the problem) r? @larsbergstrom <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14331) <!-- Reviewable:end -->
|
Updated with authless buildbot steps. The API limit is 20 requests every ~2 minutes, so this should be mostly fine. In cases where it isn't fine, this will just make us fall back to our old behavior of flunking builds on all wpt/css failures. |
|
@bors-servo r+ |
|
|
Add ./mach filter-intermittents for catching intermittents on CI cc @metajack The plan here is to run this on the error summary of the logs after each WPT/CSS run, as `./mach filter-intermittents wpt-errorsummary.log --output filtered-errorsummary.log --auth /path/to/authfile` The `filtered-errorsummary.log` file will be a buildbot artifact for this run, and can be used for updating test results. We change WPT/CSS runs to not cause test failures on Buildbot; instead; the test failure will be caused by this job. We should at some point add a separate highfive task which ccs the appropriate bugs and tracks consistent failures. (We can change mach filter-intermittents to output a second file of intermittents matched to bug numbers to make this easy) A small issue with this is that this simple task might mask failures of the WPT harness itself. We have a separate test that tests if the test harness works though. (This test will fail if the log files it requires don't exist, perhaps that's enough to solve the problem) r? @larsbergstrom <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14331) <!-- Reviewable:end -->
|
@bors-servo r- |
|
|
|
@bors-servo r=larsbergstrom,jdm |
|
|
…rom,jdm Add ./mach filter-intermittents for catching intermittents on CI cc @metajack The plan here is to run this on the error summary of the logs after each WPT/CSS run, as `./mach filter-intermittents wpt-errorsummary.log --output filtered-errorsummary.log --auth /path/to/authfile` The `filtered-errorsummary.log` file will be a buildbot artifact for this run, and can be used for updating test results. We change WPT/CSS runs to not cause test failures on Buildbot; instead; the test failure will be caused by this job. We should at some point add a separate highfive task which ccs the appropriate bugs and tracks consistent failures. (We can change mach filter-intermittents to output a second file of intermittents matched to bug numbers to make this easy) A small issue with this is that this simple task might mask failures of the WPT harness itself. We have a separate test that tests if the test harness works though. (This test will fail if the log files it requires don't exist, perhaps that's enough to solve the problem) r? @larsbergstrom <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14331) <!-- Reviewable:end -->
|
|
… uploads as an artefact
|
@bors-servo r=larsbergstrom,jdm |
|
|
…rom,jdm Add ./mach filter-intermittents for catching intermittents on CI cc @metajack The plan here is to run this on the error summary of the logs after each WPT/CSS run, as `./mach filter-intermittents wpt-errorsummary.log --output filtered-errorsummary.log --auth /path/to/authfile` The `filtered-errorsummary.log` file will be a buildbot artifact for this run, and can be used for updating test results. We change WPT/CSS runs to not cause test failures on Buildbot; instead; the test failure will be caused by this job. We should at some point add a separate highfive task which ccs the appropriate bugs and tracks consistent failures. (We can change mach filter-intermittents to output a second file of intermittents matched to bug numbers to make this easy) A small issue with this is that this simple task might mask failures of the WPT harness itself. We have a separate test that tests if the test harness works though. (This test will fail if the log files it requires don't exist, perhaps that's enough to solve the problem) r? @larsbergstrom <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14331) <!-- Reviewable:end -->
|
|
|
Looks like the builders have outdated certs. Doing a python GET on https://www.joshmatthews.net/intermittent-tracker/query.py?name=private gives an HTTP 403 error (http://build.servo.org/builders/linux-rel-css/builds/1186), or if using nfshost.com is NearlyFreeSpeech's certificate, which is probably the older one. It now uses Lets Encrypt, and it seems like Python on the builders doesn't know about this. |
|
Only on Mac I guess, since filter-intermittents worked in http://build.servo.org/builders/mac-rel-css/builds/4508 |
|
Can we host this new service on build.servo.org instead? That can be done in a followup of course. |
|
We can and should; I planned to do this once we were sure that things were working. |
Manishearth commentedNov 23, 2016
•
edited by larsbergstrom
cc @metajack
The plan here is to run this on the error summary of the logs after each WPT/CSS run, as
./mach filter-intermittents wpt-errorsummary.log --output filtered-errorsummary.log --auth /path/to/authfileThe
filtered-errorsummary.logfile will be a buildbot artifact for this run, and can be used for updating test results.We change WPT/CSS runs to not cause test failures on Buildbot; instead; the test failure will be caused by this job.
We should at some point add a separate highfive task which ccs the appropriate bugs and tracks consistent failures. (We can change mach filter-intermittents to output a second file of intermittents matched to bug numbers to make this easy)
A small issue with this is that this simple task might mask failures of the WPT harness itself. We have a separate test that tests if the test harness works though. (This test will fail if the log files it requires don't exist, perhaps that's enough to solve the problem)
r? @larsbergstrom
This change is