Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd CI script to retry builds #9572
Conversation
|
(Of course, we should only run this on test-ref. Most PRs don't change things that affect test-ref; and test-ref is where all the intermittents are from) |
|
test-ref doesn't exist. test-wpt and test-css are both sources of intermittent failures... |
|
Ugh, I meant test-css, but yeah, wpt has reftests too. Not sure if we should retry wpt since plenty of legit failures come from it too. |
|
If you're mainly concerned about test-css, I'd almost be more interested in running specific intermittent failures with non-incremental layout instead (assuming that is, indeed, the source of our bugs). I guess I'm personally just really afraid of auto-retry because it starts with this, then goes to "retry just the failed test," and then shortly people start to forget that it's even there and just have a general sense that tests are flaky and/or "only run on the builders." I'm mainly citing from past life experiences, and admittedly the Visual Studio tests drove the UI via accessibility automation, which is significantly more challenging to get reproducible than our wpt/css tests, so I might be overly worried. |
|
@bors-servo r+ I'll go ahead and land this so that we can test this out and/or bikeshed our opinions on retry over in the saltfs repo where we'd be making the concrete changes :-) |
|
|
Add CI script to retry builds We can then run test-ref as `retry.sh 2 ./mach test-ref ....` We could also have this curl to some server with information on the intermittent; so that we can keep track of these separately. It would also be nice to have crowbot report them in chat (@jdm, is this possible?) and have something comment on GH with a cross-reference. This isn't too different from what we do already, we mostly just file or link to an intermittent hit retry, and move on. r? @larsbergstrom cc @jack @edunham <!-- Reviewable:start --> [<img src="https://reviewable.io/review_button.svg" height="40" alt="Review on Reviewable"/>](https://reviewable.io/reviews/servo/servo/9572) <!-- Reviewable:end -->
|
|
Clean up CI bash scripts Also reverts #9572. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/10948) <!-- Reviewable:end -->
Manishearth commentedFeb 8, 2016
We can then run test-ref as
retry.sh 2 ./mach test-ref ....We could also have this curl to some server with information on the
intermittent; so that we can keep track of these separately. It would also be
nice to have crowbot report them in chat (@jdm, is this possible?) and have
something comment on GH with a cross-reference.
This isn't too different from what we do already, we mostly just file or link to
an intermittent hit retry, and move on.
r? @larsbergstrom
cc @jack @edunham