Run a couple WPT tests on Android on CI

[SimonSapin]

This change is 

[highfive]

Heads up! This PR modifies the following files:

• @aneeshusa: etc/ci/buildbot_steps.yml

[SimonSapin]

@bors-servo try

[bors-servo]

⌛ Trying commit fac9acc with merge 7c4cce3...

[bors-servo]

💔 Test failed - android-x86

[highfive]

  ▶ TIMEOUT [expected OK] /_mozilla/mozilla/DOMParser.html

  ▶ TIMEOUT [expected OK] /_mozilla/mozilla/webgl/context_creation_error.html

[SimonSapin]

@bors-servo try

[bors-servo]

⌛ Trying commit c99c93f with merge 90e1a38...

[bors-servo]

💔 Test failed - android-x86

[SimonSapin]

 0:37.72 TEST_START: /_mozilla/mozilla/DOMParser.html
 0:39.23 pid:4903 Entering android_main
 0:39.23 pid:4903 Creating application thread
 0:39.23 pid:4903 ERROR 2018-07-24T21:32:43Z: script::dom::bindings::error: Error at :2:1 timeout is not defined
 0:53.27 TEST_END: TIMEOUT, expected OK

So far I did not reproduce this failure on my linux machine. I think that the undefined JS variable is the first argument in this call:

servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/testharness_servodriver.js

Line 2 in 65df9cb

window.__wd_results_timer__ = setTimeout(timeout, %(timeout)s);

… and that it is supposed to be defined here:

servo/tests/wpt/web-platform-tests/resources/testharness.js

Lines 2232 to 2237 in 65df9cb

	function timeout() {
	if (tests.timeout_length === null) {
	tests.timeout();
	}
	}
	expose(timeout, 'timeout');

… with expose() effectively doing global_scope["timeout"] = timeout.

The former script is injected through WebDriver here:

servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/executors/executorservodriver.py

Lines 220 to 225 in 65df9cb

	result = json.loads(
	session.execute_async_script(
	self.script % {"abs_url": url,
	"url": strip_server(url),
	"timeout_multiplier": self.timeout_multiplier,
	"timeout": timeout * 1000}))

The latter is part of every test like this:

servo/tests/wpt/mozilla/tests/mozilla/DOMParser.html

Line 3 in 234951e

<script src="/resources/testharness.js"></script>

[SimonSapin]

Although this issue does not reproduce on my linux desktop (mach test-wpt-android /_mozilla/mozilla/DOMParser.html --release succeeds), it does reproduce on my linux laptop and on my macbook. Maybe it is timing-related (the desktop has a faster CPU), but adding time.sleep(1) before the execute_async_script(…) call does not help.

@jdm I’m running out of leads here, I don’t know what to try next.

[jdm]

The code you linked seems like a reasonable lead. What about debug output to show what code is actually executing?

[SimonSapin]

Mystery solved.

I was assuming that there would be no issue with the test loading /resources/testharness.js, but sprinkling some console.log(something) lines here and there showed that this is not happening at all on machines where I observe the issue. The test is not loading at all!

wptrunner asks Servo (in this case through WebDriver) to load URLs like http://web-platform.test:8000/_mozilla/mozilla/DOMParser.html. It also generates a temporary file with a syntax compatible with /etc/hosts that maps the hostname web-platform.test and some others to 127.0.01, and sets the HOST_FILE environment variable to tell Servo where to find it. However in the case of the Android emulator, the host file does not exist in the environment’s filesystem and environment variables are not propagated. So Servo gets a DNS error while trying to resolve the hostname web-platform.test and the document load fails. Additionally, because of #21248, that failure is not propagated back to the WebDriver client. The WebDriver command appears to succeed, so wptrunner carries on and only later times out when test results do not arrive.

As to what happens on my Linux desktop machine, DNS resolution in the Android emulator appears to be delegated to the host. And web-platform.test is valid on this machine because I had modified /etc/hosts by hand weeks ago when wptserve asked me to do so when I tried to use it without mach.

So what’s needed is, most likely:

• Adding a mechanism similar to /sdcard/Android/data/com.mozilla.servo/files/android_params to allow setting environment variables in the Servo process on Android. (With code at startup that reads a file and calls std::env::set_var.)
• Add support in etc/run_in_headless_android_emulator.py for the HOST_FILE env variable, similar to existing support for --user-stylesheet: use adb push to make the file available in the emulator’s filesystem, and then use new mechanism to propagate the env variable

[SimonSapin]

I ended up adding not a general mechanism for env variables, but a special case for /sdcard/Android/data/com.mozilla.servo/files/android_hosts.

I’ve tested this through SSH on servo-linux-kvm1, this should be ready to land.

[nox]

@bors-servo r+

[bors-servo]

📌 Commit d3107a3 has been approved by nox

[jdm]

@bors-servo retry

• #13480

[bors-servo]

⌛ Testing commit 5e5b89a with merge b94fd23...

[bors-servo]

💔 Test failed - linux-rel-wpt

[jdm]

@bors-servo retry

• #10656

[bors-servo]

⚡ Previous build results for android, android-x86, arm32, linux-dev, linux-rel-css, mac-dev-unit, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, windows-msvc-dev are reusable. Rebuilding only arm64, linux-rel-wpt...

[bors-servo]

💔 Test failed - linux-rel-wpt

[jdm]

@bors-servo retry

• #21067

[bors-servo]

⌛ Testing commit 5e5b89a with merge e9bc6f4...

[bors-servo]

💔 Test failed - mac-rel-css2

[jdm]

@bors-servo retry

• Websockets

[bors-servo]

⚡ Previous build results for android, android-x86, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css1, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, windows-msvc-dev are reusable. Rebuilding only mac-rel-css2...

[bors-servo]

☀️ Test successful - android, android-x86, arm32, arm64, linux-dev, linux-rel-css, linux-rel-wpt, mac-dev-unit, mac-rel-css1, mac-rel-css2, mac-rel-wpt1, mac-rel-wpt2, mac-rel-wpt3, mac-rel-wpt4, windows-msvc-dev
Approved by: jdm
Pushing e9bc6f4 to master...