Update to latest wptrunner

tests/wpt/harness/README.rst

@@ -29,7 +29,7 @@ following are most significant:
   The path to a binary file for the product (browser) to test against.
 
 ``--webdriver-binary`` (required if product is `chrome`)
-  The path to a `*driver` binary; e.g., a `chromedriver` binary.
+  The path to a `driver` binary; e.g., a `chromedriver` binary.
 
 ``--certutil-binary`` (required if product is `firefox` [#]_)
   The path to a `certutil` binary (for tests that must be run over https).
@@ -43,13 +43,18 @@ following are most significant:
 ``--prefs-root`` (required only when testing a Firefox binary)
   The path to a directory containing Firefox test-harness preferences. [#]_
 
+``--config`` (should default to `wptrunner.default.ini`)
+  The path to the config (ini) file.
+
 .. [#] The ``--certutil-binary`` option is required when the product is
    ``firefox`` unless ``--ssl-type=none`` is specified.
 
 .. [#] The ``--metadata`` path is to a directory that contains:
 
-  * a ``MANIFEST.json`` file (the web-platform-tests documentation has
-    instructions on generating this file); and
+  * a ``MANIFEST.json`` file (instructions on generating this file are
+    available in the `detailed documentation
+    <http://wptrunner.readthedocs.org/en/latest/usage.html#installing-wptrunner>`_);
+    and
   * (optionally) any expectation files (see below)
 
 .. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
@@ -125,7 +130,7 @@ input to the `wptupdate` tool.
 Expectation File Format
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-Metadat about tests, notably including their expected results, is
+Metadata about tests, notably including their expected results, is
 stored in a modified ini-like format that is designed to be human
 editable, but also to be machine updatable.
 

tests/wpt/harness/docs/usage.rst

@@ -28,19 +28,19 @@ environment created as above::
   pip install -e ./
 
 In addition to the dependencies installed by pip, wptrunner requires
-a copy of the web-platform-tests repository. That can be located
-anywhere on the filesystem, but the easiest option is to put it within
-the wptrunner checkout directory, as a subdirectory named ``tests``::
+a copy of the web-platform-tests repository. This can be located
+anywhere on the filesystem, but the easiest option is to put it
+under the same parent directory as the wptrunner checkout::
 
-  git clone https://github.com/w3c/web-platform-tests.git tests
+  git clone https://github.com/w3c/web-platform-tests.git
 
 It is also necessary to generate a web-platform-tests ``MANIFEST.json``
-file. It's recommended to put that within the wptrunner
-checkout directory, in a subdirectory named ``meta``::
+file. It's recommended to also put that under the same parent directory as
+the wptrunner checkout, in a directory named ``meta``::
 
   mkdir meta
-  cd tests
-  python tools/scripts/manifest.py ../meta/MANIFEST.json
+  cd web-platform-tests
+  python manifest --path ../meta/MANIFEST.json
 
 The ``MANIFEST.json`` file needs to be regenerated each time the
 web-platform-tests checkout is updated. To aid with the update process
@@ -74,6 +74,9 @@ takes multiple options, of which the following are most significant:
 ``--prefs-root`` (required only when testing a Firefox binary)
   The path to a directory containing Firefox test-harness preferences. [#]_
 
+``--config`` (should default to `wptrunner.default.ini`)
+  The path to the config (ini) file.
+
 .. [#] The ``--certutil-binary`` option is required when the product is
    ``firefox`` unless ``--ssl-type=none`` is specified.
 
@@ -94,10 +97,17 @@ The following examples show how to start wptrunner with various options.
 Starting wptrunner
 ------------------
 
+The examples below assume the following directory layout,
+though no specific folder structure is required::
+
+  ~/testtwf/wptrunner          # wptrunner checkout
+  ~/testtwf/web-platform-tests # web-platform-tests checkout
+  ~/testtwf/meta               # metadata
+
 To test a Firefox Nightly build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
-  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
     --binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
     --certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
     --prefs-root=~/mozilla-central/testing/profiles
@@ -106,7 +116,7 @@ wptrunner using something similar to the following example::
 And to test a Chromium build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
-  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
     --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
     --webdriver-binary=/usr/local/bin/chromedriver --product=chrome
 
@@ -118,7 +128,7 @@ To restrict a test run just to tests in a particular web-platform-tests
 subdirectory, specify the directory name in the positional arguments after
 the options; for example, run just the tests in the `dom` subdirectory::
 
-  wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
+  wptrunner --metadata=~/testtwf/meta --tests=~/testtwf/web-platform-tests/ \
     --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
     --prefs-root=/path/to/testing/profiles \
     dom
@@ -180,7 +190,7 @@ Configuration File
 
 wptrunner uses a ``.ini`` file to control some configuration
 sections. The file has three sections; ``[products]``,
-``[paths]`` and ``[web-platform-tests]``.
+``[manifest:default]`` and ``[web-platform-tests]``.
 
 ``[products]`` is used to
 define the set of available products. By default this section is empty
@@ -195,12 +205,12 @@ e.g.::
   chrome =
   netscape4 = path/to/netscape.py
 
-``[paths]`` specifies the default paths for the tests and metadata,
+``[manifest:default]`` specifies the default paths for the tests and metadata,
 relative to the config file. For example::
 
-  [paths]
-  tests = checkouts/web-platform-tests
-  metadata = /home/example/wpt/metadata
+  [manifest:default]
+  tests = ~/testtwf/web-platform-tests
+  metadata = ~/testtwf/meta
 
 
 ``[web-platform-tests]`` is used to set the properties of the upstream

tests/wpt/harness/wptrunner/browsers/b2g.py

@@ -192,7 +192,7 @@ def __init__(self, *args, **kwargs):
 
         import sys, subprocess
 
-        self.device = mozdevice.ADBDevice()
+        self.device = mozdevice.ADBB2G()
         self.device.forward("tcp:%s" % self.marionette_port,
                             "tcp:2828")
         self.executor = None

tests/wpt/harness/wptrunner/browsers/firefox.py

@@ -28,7 +28,8 @@
                  "browser_kwargs": "browser_kwargs",
                  "executor_kwargs": "executor_kwargs",
                  "env_options": "env_options",
-                 "run_info_extras": "run_info_extras"}
+                 "run_info_extras": "run_info_extras",
+                 "update_properties": "update_properties"}
 
 
 def check_args(**kwargs):
@@ -54,7 +55,7 @@ def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
                                            cache_manager, **kwargs)
     executor_kwargs["close_after_done"] = True
     if kwargs["timeout_multiplier"] is None:
-        if kwargs["gecko_e10s"] and test_type == "reftest":
+        if test_type == "reftest":
             if run_info_data["debug"]:
                 executor_kwargs["timeout_multiplier"] = 4
             else:
@@ -71,9 +72,14 @@ def env_options():
             "certificate_domain": "web-platform.test",
             "supports_debugger": True}
 
+
 def run_info_extras(**kwargs):
     return {"e10s": kwargs["gecko_e10s"]}
 
+
+def update_properties():
+    return ["debug", "e10s", "os", "version", "processor", "bits"], {"debug", "e10s"}
+
 class FirefoxBrowser(Browser):
     used_ports = set()
 

tests/wpt/harness/wptrunner/browsers/servo.py

@@ -17,7 +17,9 @@
                               "reftest": "ServoRefTestExecutor"},
                  "browser_kwargs": "browser_kwargs",
                  "executor_kwargs": "executor_kwargs",
-                 "env_options": "env_options"}
+                 "env_options": "env_options",
+                 "run_info_extras": "run_info_extras",
+                 "update_properties": "update_properties"}
 
 
 def check_args(**kwargs):
@@ -47,8 +49,16 @@ def env_options():
             "supports_debugger": True}
 
 
+def run_info_extras(**kwargs):
+    return {"backend": kwargs["servo_backend"]}
+
+
+def update_properties():
+    return ["debug", "os", "version", "processor", "bits", "backend"], None
+
+
 def render_arg(render_backend):
-    return {"cpu": "--cpu"}[render_backend]
+    return {"cpu": "--cpu", "webrender": "--webrender"}[render_backend]
 
 
 class ServoBrowser(NullBrowser):

tests/wpt/harness/wptrunner/browsers/servodriver.py

@@ -23,7 +23,9 @@
                               "reftest": "ServoWebDriverRefTestExecutor"},
                  "browser_kwargs": "browser_kwargs",
                  "executor_kwargs": "executor_kwargs",
-                 "env_options": "env_options"}
+                 "env_options": "env_options",
+                 "run_info_extras": "run_info_extras",
+                 "update_properties": "update_properties"}
 
 hosts_text = """127.0.0.1 web-platform.test
 127.0.0.1 www.web-platform.test
@@ -59,6 +61,14 @@ def env_options():
             "supports_debugger": True}
 
 
+def run_info_extras(**kwargs):
+    return {"backend": kwargs["servo_backend"]}
+
+
+def update_properties():
+    return ["debug", "os", "version", "processor", "bits", "backend"], None
+
+
 def make_hosts_file():
     hosts_fd, hosts_path = tempfile.mkstemp()
     with os.fdopen(hosts_fd, "w") as f:
@@ -88,6 +98,7 @@ def start(self):
 
         env = os.environ.copy()
         env["HOST_FILE"] = self.hosts_path
+        env["RUST_BACKTRACE"] = "1"
 
         debug_args, command = browser_command(self.binary,
                                               [render_arg(self.render_backend), "--hard-fail",

tests/wpt/harness/wptrunner/executors/executormarionette.py

@@ -107,12 +107,6 @@ def is_alive(self):
         return True
 
     def after_connect(self):
-        # Turn off debug-level logging by default since this is so verbose
-        with self.marionette.using_context("chrome"):
-            self.marionette.execute_script("""
-              Components.utils.import("resource://gre/modules/Log.jsm");
-              Log.repository.getLogger("Marionette").level = Log.Level.Info;
-            """)
         self.load_runner("http")
 
     def load_runner(self, protocol):

tests/wpt/harness/wptrunner/executors/executorservo.py

@@ -87,7 +87,7 @@ def do_test(self, test):
 
         env = os.environ.copy()
         env["HOST_FILE"] = self.hosts_path
-
+        env["RUST_BACKTRACE"] = "1"
 
 
         if not self.interactive:
@@ -223,6 +223,7 @@ def screenshot(self, test, viewport_size, dpi):
 
             env = os.environ.copy()
             env["HOST_FILE"] = self.hosts_path
+            env["RUST_BACKTRACE"] = "1"
 
             if not self.interactive:
                 self.proc = ProcessHandler(self.command,

tests/wpt/harness/wptrunner/manifestinclude.py

@@ -8,6 +8,7 @@
 representing the file and each subnode representing a subdirectory that should
 be included or excluded.
 """
+import glob
 import os
 import urlparse
 
@@ -90,29 +91,36 @@ def _add_rule(self, test_manifests, url, direction):
             variant += "?" + query
 
         maybe_path = os.path.join(rest, last)
-
-        if os.path.exists(maybe_path):
-            for manifest, data in test_manifests.iteritems():
-                rel_path = os.path.relpath(maybe_path, data["tests_path"])
-                if ".." not in rel_path.split(os.sep):
-                    url = data["url_base"] + rel_path.replace(os.path.sep, "/") + variant
-                    break
+        paths = glob.glob(maybe_path)
+
+        if paths:
+            urls = []
+            for path in paths:
+                for manifest, data in test_manifests.iteritems():
+                    rel_path = os.path.relpath(path, data["tests_path"])
+                    if ".." not in rel_path.split(os.sep):
+                        urls.append(data["url_base"] + rel_path.replace(os.path.sep, "/") + variant)
+                        break
+        else:
+            urls = [url]
 
         assert direction in ("include", "exclude")
-        components = self._get_components(url)
 
-        node = self
-        while components:
-            component = components.pop()
-            if component not in node.child_map:
-                new_node = IncludeManifest(DataNode(component))
-                node.append(new_node)
-                new_node.set("skip", node.get("skip", {}))
+        for url in urls:
+            components = self._get_components(url)
+
+            node = self
+            while components:
+                component = components.pop()
+                if component not in node.child_map:
+                    new_node = IncludeManifest(DataNode(component))
+                    node.append(new_node)
+                    new_node.set("skip", node.get("skip", {}))
 
-            node = node.child_map[component]
+                node = node.child_map[component]
 
-        skip = False if direction == "include" else True
-        node.set("skip", str(skip))
+            skip = False if direction == "include" else True
+            node.set("skip", str(skip))
 
     def add_include(self, test_manifests, url_prefix):
         """Add a rule indicating that tests under a url path