Added /etc/ci/chaos_monkey_test.py. #10899

asajeffrey · 2016-04-28T16:56:25Z

Add a test to /etc/ci which runs a subset of test-wpt with --random-pipeline-failure-probability=0.2, and checks to make sure that there's no CRASH reports, so the constellation survived the experience, even if a lot of tests failed.

IRC conversation at http://logs.glob.uno/?c=mozilla%23servo&s=27+Apr+2016&e=27+Apr+2016#c416510

Fixes #10568.

r? @aneeshusa

This change is

asajeffrey · 2016-04-28T19:21:24Z

Fixes #10568

frewsxcv · 2016-04-28T19:24:19Z

etc/ci/chaos_monkey_test.py

+
+for line in TEST_RESULTS.stdout:
+    report = json.loads(line.decode('utf-8'))
+    if report.get("action", None) == "process_output":


The default value for dict.get is None, so explicitly specifying isn't necessary

frewsxcv · 2016-04-28T19:26:01Z

etc/ci/chaos_monkey_test.py

+        TEST_CRASHES = TEST_CRASHES or (status == "CRASH")
+
+if TEST_CRASHES:
+    exit(1)


You probably want sys.exit

asajeffrey · 2016-04-28T19:34:30Z

Fixed and squashed. You can tell my python is a bit rusty.

aneeshusa · 2016-04-28T21:38:55Z

etc/ci/chaos_monkey_test.py

+from subprocess import Popen, PIPE
+
+
+def is_crash(report):


This method isn't being used, so it can be removed.

aneeshusa · 2016-04-28T21:39:34Z

etc/ci/chaos_monkey_test.py

+
+TEST_RESULTS = Popen(TEST_CMD, stdout=PIPE)
+TEST_CRASHES = False
+TEST_STDOUT = {}


None of these three are constants, so they should be in lowercase camel_case.

aneeshusa · 2016-04-28T21:42:34Z

etc/ci/chaos_monkey_test.py

+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+import json


Let's add from __future__ import absolute_import, print_function above this (with an empty line between that and import json).

aneeshusa · 2016-04-28T21:46:06Z

etc/ci/chaos_monkey_test.py

+for line in TEST_RESULTS.stdout:
+    report = json.loads(line.decode('utf-8'))
+    if report.get("action") == "process_output":
+        print(report.get("thread") + " - " + report.get("data"))


Let's use str.format to make these nicer:

print("{} - {}".format(report.get('thread'), report.get('data')))

aneeshusa · 2016-04-28T21:46:19Z

etc/ci/chaos_monkey_test.py

+        print(report.get("thread") + " - " + report.get("data"))
+    status = report.get("status")
+    if status:
+        print(report.get("thread") + " - " + status + " - " + report.get("test"))


Ditto about using str.format.

aneeshusa · 2016-04-28T21:50:45Z

Which outputs of this do you need? We can gate on the exit code, and Buildbot will automatically pick up on anything we print() to stdout. If you want to capture the chaos-monkey.log, we'll need to make that explicit when adding it to steps.yml (or send it to stderr, which should get picked up automatically).

I looked at the output and there's quite a bit of it - do we only want to log the CRASHes to output, or is having the rest of it useful as well?

aneeshusa · 2016-04-28T21:52:54Z

etc/ci/chaos_monkey_test.py

+    status = report.get("status")
+    if status:
+        print(report.get("thread") + " - " + status + " - " + report.get("test"))
+        TEST_CRASHES = TEST_CRASHES or (status == "CRASH")


style nit: I find this easier to read, but up to you:

if status == "CRASH": TEST_CRASHES = True

aneeshusa · 2016-04-28T21:54:42Z

Will we ever have both action == 'process_output' and a status? Not sure if you want to avoid printing two lines in that case.

asajeffrey · 2016-04-28T22:39:15Z

IRC chat about outputs: http://logs.glob.uno/?c=mozilla%23servo&s=28+Apr+2016&e=28+Apr+2016#c417879

TL;DR: don't capture the whole log.

asajeffrey · 2016-04-28T22:40:50Z

action and status are exclusive, even if we do get a line with both, it's worth printing out both lots of info.

asajeffrey · 2016-04-28T23:05:09Z

Fixed and squashed.

asajeffrey · 2016-04-28T23:05:45Z

I also increased the probability of pipeline closure to 0.1, to make sure that some crashes happen! This magic number may need played with.

aneeshusa · 2016-04-28T23:16:27Z

etc/ci/chaos_monkey_test.py

+# returned by the test command (which is why we can't use check_output).
+
+test_results = Popen(TEST_CMD, stdout=PIPE)
+test_crashes = False


Quick nit: Let's change this to any_crashes because if any_crashes reads better.

Sure. I'll do this tomorrow, when I'm at a computer.


        Added /etc/ci/chaos_monkey_test.py.

asajeffrey · 2016-04-29T14:58:03Z

Fixed and squashed.

aneeshusa · 2016-04-29T15:26:22Z

@bors-servo r+

We should file a follow-up issue for gating on this in the CI.

bors-servo · 2016-04-29T15:26:23Z

📌 Commit ac03728 has been approved by aneeshusa

asajeffrey · 2016-04-29T16:15:44Z

Filed #10927

bors-servo · 2016-04-29T23:01:46Z

⌛ Testing commit ac03728 with merge a7a8eba...


        Auto merge of #10899 - asajeffrey:chaos-monkey-ci, r=aneeshusa

@aneeshusa

Added /etc/ci/chaos_monkey_test.py. Add a test to `/etc/ci` which runs a subset of `test-wpt` with `--random-pipeline-failure-probability=0.2`, and checks to make sure that there's no `CRASH` reports, so the constellation survived the experience, even if a lot of tests failed. IRC conversation at http://logs.glob.uno/?c=mozilla%23servo&s=27+Apr+2016&e=27+Apr+2016#c416510 Fixes #10568. r? @aneeshusa  --- This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/10899)

bors-servo · 2016-04-30T00:18:56Z

☀️ Test successful - android, arm32, arm64, linux-dev, linux-rel, mac-dev-unit, mac-rel-css, mac-rel-wpt, windows

highfive assigned aneeshusa Apr 28, 2016

highfive added the S-awaiting-review label Apr 28, 2016

asajeffrey mentioned this pull request Apr 28, 2016

Constellation hardening metabug #10124

Closed

frewsxcv reviewed Apr 28, 2016
View changes

etc/ci/chaos_monkey_test.py Outdated

TEST_CRASHES = TEST_CRASHES or (status == "CRASH")

if TEST_CRASHES:

exit(1)

This comment has been minimized.

Sign in to view

Copy link

frewsxcv Apr 28, 2016
Member

You probably want sys.exit

asajeffrey force-pushed the asajeffrey:chaos-monkey-ci branch from dfafabc to 118126f Apr 28, 2016

aneeshusa reviewed Apr 28, 2016
View changes

asajeffrey force-pushed the asajeffrey:chaos-monkey-ci branch from 118126f to ee8a4ad Apr 28, 2016

aneeshusa reviewed Apr 28, 2016
View changes

Added /etc/ci/chaos_monkey_test.py.

Loading status checks…

ac03728

asajeffrey force-pushed the asajeffrey:chaos-monkey-ci branch from ee8a4ad to ac03728 Apr 29, 2016

highfive added S-awaiting-merge and removed S-awaiting-review labels Apr 29, 2016

asajeffrey mentioned this pull request Apr 29, 2016

Gate on /etc/ci/chaos_monkey_test.py #10927

Open

bors-servo mentioned this pull request Apr 29, 2016

Change OSX 'activiation policy' when running headless mode. #10935

Merged

bors-servo merged commit ac03728 into servo:master Apr 30, 2016
3 checks passed

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

highfive removed the S-awaiting-merge label Apr 30, 2016

servo / servo

Added /etc/ci/chaos_monkey_test.py. #10899

Added /etc/ci/chaos_monkey_test.py. #10899

asajeffrey commented Apr 28, 2016 •

edited by KiChjang

asajeffrey commented Apr 28, 2016

This comment has been minimized.

This comment has been minimized.

asajeffrey commented Apr 28, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Apr 28, 2016 •

edited by larsbergstrom

This comment has been minimized.

aneeshusa commented Apr 28, 2016

asajeffrey commented Apr 28, 2016 •

edited by larsbergstrom

asajeffrey commented Apr 28, 2016

asajeffrey commented Apr 28, 2016

asajeffrey commented Apr 28, 2016 •

edited

This comment has been minimized.

This comment has been minimized.

asajeffrey commented Apr 29, 2016

aneeshusa commented Apr 29, 2016

bors-servo commented Apr 29, 2016

asajeffrey commented Apr 29, 2016

bors-servo commented Apr 29, 2016

bors-servo commented Apr 30, 2016

servo / servo

Join GitHub today

Added /etc/ci/chaos_monkey_test.py. #10899

Added /etc/ci/chaos_monkey_test.py. #10899

Conversation

asajeffrey commented Apr 28, 2016 • edited by KiChjang

asajeffrey commented Apr 28, 2016

This comment has been minimized.

This comment has been minimized.

asajeffrey commented Apr 28, 2016

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa Apr 28, 2016 • edited

This comment has been minimized.

This comment has been minimized.

aneeshusa Apr 28, 2016 • edited by larsbergstrom

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aneeshusa commented Apr 28, 2016 • edited by larsbergstrom

This comment has been minimized.

aneeshusa Apr 28, 2016 • edited by larsbergstrom

aneeshusa commented Apr 28, 2016

asajeffrey commented Apr 28, 2016 • edited by larsbergstrom

asajeffrey commented Apr 28, 2016

asajeffrey commented Apr 28, 2016

asajeffrey commented Apr 28, 2016 • edited

This comment has been minimized.

This comment has been minimized.

asajeffrey commented Apr 29, 2016

aneeshusa commented Apr 29, 2016

bors-servo commented Apr 29, 2016

asajeffrey commented Apr 29, 2016

bors-servo commented Apr 29, 2016

bors-servo commented Apr 30, 2016

asajeffrey commented Apr 28, 2016 •

edited by KiChjang

aneeshusa Apr 28, 2016 •

edited

aneeshusa Apr 28, 2016 •

edited by larsbergstrom

aneeshusa commented Apr 28, 2016 •

edited by larsbergstrom

aneeshusa Apr 28, 2016 •

edited by larsbergstrom

asajeffrey commented Apr 28, 2016 •

edited by larsbergstrom

asajeffrey commented Apr 28, 2016 •

edited