Intermittents #12629

notriddle · 2016-07-28T00:44:05Z

We probably want to have Buildbot run that script.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix #12435, #11574, and #11561
These changes do not require tests because reasons

This change is

highfive · 2016-07-28T00:44:09Z

Heads up! This PR modifies the following files:

@aneeshusa: etc/ci/buildbot_steps.yml, etc/ci/check_intermittents.sh, etc/ci/former_intermittents_wpt.txt, etc/ci/former_intermittents_css.txt

jdm · 2016-07-28T11:34:43Z

r? @aneeshusa

wafflespeanut · 2016-07-29T06:41:51Z

etc/ci/check_intermittents.sh

+
+for test_type in wpt css; do
+    while read test_name; do
+        echo "  - Checking $test_name"


Maybe a better message? like, "Checking $test_name because it's a known intermittent..."

It's inside a script called check_intermittents, and every CI system in the world shows which command was run before its actual stdout. The only reason I included - Checking was to help distinguish it from the output of wptrunner itself.

wafflespeanut · 2016-07-29T06:42:14Z

This is a great idea! 👍

aneeshusa · 2016-08-04T15:49:15Z

etc/ci/buildbot_steps.yml

@@ -27,6 +27,14 @@ mac-nightly:
  - ./mach package --release
  - ./etc/ci/upload_nightly.sh mac

+linux-intermittent:


CI doesn't read from this file yet, it's currently reading from buildbot/master/files/config/steps.yml file in the saltfs repo, which is where we'll need to make this change.

Also, let's name these linux-rel-intermittent and mac-rel-intermittent for consistency.

aneeshusa · 2016-08-04T15:50:22Z

etc/ci/check_intermittents.sh

@@ -0,0 +1,14 @@
+#!/bin/sh


Shebang should be #!/usr/bin/env bash per the style guide.

Also, please add the errexit, nounset, and pipefail options.


        Call intermittent checker

Part of servo/servo#12629

aneeshusa · 2016-08-04T16:11:41Z

etc/ci/check_intermittents.sh

+for test_type in wpt css; do
+    while read test_name; do
+        echo "  - Checking $test_name"
+        ./mach test-${test_type} --release --repeat $REPEAT_COUNT "$test_name" > tmp_log.txt


Let's append to the log file always and have Buildbot handle showing the log (per this comment). Also, please use the full form and quotes "${some_variable}" for the variables (test_type and REPEAT_COUNT).

aneeshusa · 2016-08-04T16:12:03Z

etc/ci/check_intermittents.sh

+    while read test_name; do
+        echo "  - Checking $test_name"
+        ./mach test-${test_type} --release --repeat $REPEAT_COUNT "$test_name" > tmp_log.txt
+        if [ $? != 0 ]; then


Let set -o errexit handle the short circuit exiting.

aneeshusa · 2016-08-04T16:12:19Z

etc/ci/check_intermittents.sh

+            cat tmp_log.txt
+            exit $?
+        fi
+    done < etc/ci/former_intermittents_${test_type}.txt


Please quote the filename due to the variable substitution.

aneeshusa · 2016-08-04T16:12:28Z

etc/ci/check_intermittents.sh

+        fi
+    done < etc/ci/former_intermittents_${test_type}.txt
+done
+


nit: remove the empty extra newline.

aneeshusa · 2016-08-09T16:55:33Z

etc/ci/check_intermittents.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+REPEAT_COUNT=100


Please put this after we set the bash options (e.g. errexit).

aneeshusa · 2016-08-09T16:56:20Z

LGTM aside from the one nit, please also rebase on a new master (to pull in some new shell lints) and squash the fixups.


        Re-enable intermittents fixed by #12543

* Fixes #11574 * Fixes #11561

notriddle · 2016-08-09T21:12:56Z

Rebased and moved.

aneeshusa · 2016-08-09T21:44:08Z

Please fix the tidy failures.


        Add script to check for intermittent regressions

Fixes #12435

aneeshusa · 2016-08-10T04:28:15Z

@bors-servo r+

bors-servo · 2016-08-10T04:28:16Z

📌 Commit c71cb19 has been approved by aneeshusa

bors-servo · 2016-08-10T04:28:18Z

⌛ Testing commit c71cb19 with merge 944cbbf...


        Auto merge of #12629 - notriddle:intermittents, r=aneeshusa

Intermittents We probably want to have Buildbot run that script. --- - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix #12435, #11574, and #11561 - [X] These changes do not require tests because reasons  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12629)


        Call intermittent checker

Part of servo/servo#12629

bors-servo · 2016-08-10T05:25:17Z

☀️ Test successful - arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows-dev


        Auto merge of #453 - notriddle:intermittents, r=aneeshusa

Call intermittent checker Part of servo/servo#12629  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/453)


        Auto merge of #453 - notriddle:intermittents, r=aneeshusa

Call intermittent checker Part of servo/servo#12629  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/453)


        Auto merge of #453 - notriddle:intermittents, r=aneeshusa

Call intermittent checker Part of servo/servo#12629  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/saltfs/453)

highfive assigned cbrewster Jul 28, 2016

highfive added the S-awaiting-review label Jul 28, 2016

highfive assigned aneeshusa and unassigned cbrewster Jul 28, 2016

wafflespeanut reviewed Jul 29, 2016
View changes

aneeshusa reviewed Aug 4, 2016
View changes

notriddle added a commit to notriddle/saltfs that referenced this pull request Aug 4, 2016

Call intermittent checker

ad12c67

Part of servo/servo#12629

notriddle mentioned this pull request Aug 4, 2016

Call intermittent checker servo/saltfs#453

Merged

aneeshusa reviewed Aug 4, 2016
View changes

etc/ci/check_intermittents.sh Outdated

cat tmp_log.txt

exit $?

fi

done < etc/ci/former_intermittents_${test_type}.txt

This comment has been minimized.

Sign in to view

aneeshusa Aug 4, 2016
Member

Please quote the filename due to the variable substitution.

aneeshusa reviewed Aug 4, 2016
View changes

etc/ci/check_intermittents.sh Outdated

fi

done < etc/ci/former_intermittents_${test_type}.txt

done

This comment has been minimized.

Sign in to view

aneeshusa Aug 4, 2016
Member

nit: remove the empty extra newline.

aneeshusa reviewed Aug 9, 2016
View changes

etc/ci/check_intermittents.sh Outdated

@@ -0,0 +1,13 @@

#!/usr/bin/env bash

REPEAT_COUNT=100

This comment has been minimized.

Sign in to view

aneeshusa Aug 9, 2016
Member

Please put this after we set the bash options (e.g. errexit).

aneeshusa added S-needs-code-changes and removed S-awaiting-review labels Aug 9, 2016

Re-enable intermittents fixed by #12543

c2be127

* Fixes #11574 * Fixes #11561

highfive added S-awaiting-review and removed S-needs-code-changes labels Aug 9, 2016

aneeshusa added S-fails-tidy and removed S-awaiting-review labels Aug 9, 2016

Add script to check for intermittent regressions

Loading status checks…

c71cb19

Fixes #12435

highfive added the S-awaiting-review label Aug 10, 2016

highfive added S-awaiting-merge and removed S-awaiting-review labels Aug 10, 2016

aneeshusa removed the S-fails-tidy label Aug 10, 2016

notriddle added a commit to notriddle/saltfs that referenced this pull request Aug 10, 2016

Call intermittent checker

21d29e9

Part of servo/servo#12629

highfive removed the S-awaiting-merge label Aug 10, 2016

servo / servo

Intermittents #12629

Intermittents #12629

notriddle commented Jul 28, 2016 •

edited by larsbergstrom

highfive commented Jul 28, 2016

jdm commented Jul 28, 2016

This comment has been minimized.

This comment has been minimized.

wafflespeanut commented Jul 29, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Aug 9, 2016

notriddle commented Aug 9, 2016

aneeshusa commented Aug 9, 2016

aneeshusa commented Aug 10, 2016

bors-servo commented Aug 10, 2016

bors-servo commented Aug 10, 2016

bors-servo commented Aug 10, 2016

servo / servo

Join GitHub today

Intermittents #12629

Intermittents #12629

Conversation

notriddle commented Jul 28, 2016 • edited by larsbergstrom

highfive commented Jul 28, 2016

jdm commented Jul 28, 2016

This comment has been minimized.

This comment has been minimized.

wafflespeanut commented Jul 29, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Aug 9, 2016

notriddle commented Aug 9, 2016

aneeshusa commented Aug 9, 2016

aneeshusa commented Aug 10, 2016

bors-servo commented Aug 10, 2016

bors-servo commented Aug 10, 2016

bors-servo commented Aug 10, 2016

notriddle commented Jul 28, 2016 •

edited by larsbergstrom