Skip to content
Permalink
Browse files

Update to latest wptrunner

  • Loading branch information
jgraham committed Feb 2, 2016
1 parent 9baa59a commit f234d99ac1dcdbfb96eaceec6726c113f7c8eea8
@@ -29,7 +29,7 @@ following are most significant:
The path to a binary file for the product (browser) to test against.

``--webdriver-binary`` (required if product is `chrome`)
The path to a `*driver` binary; e.g., a `chromedriver` binary.
The path to a `driver` binary; e.g., a `chromedriver` binary.

``--certutil-binary`` (required if product is `firefox` [#]_)
The path to a `certutil` binary (for tests that must be run over https).
@@ -43,13 +43,18 @@ following are most significant:
``--prefs-root`` (required only when testing a Firefox binary)
The path to a directory containing Firefox test-harness preferences. [#]_

``--config`` (should default to `wptrunner.default.ini`)
The path to the config (ini) file.

.. [#] The ``--certutil-binary`` option is required when the product is
``firefox`` unless ``--ssl-type=none`` is specified.
.. [#] The ``--metadata`` path is to a directory that contains:
* a ``MANIFEST.json`` file (the web-platform-tests documentation has
instructions on generating this file); and
* a ``MANIFEST.json`` file (instructions on generating this file are
available in the `detailed documentation
<http://wptrunner.readthedocs.org/en/latest/usage.html#installing-wptrunner>`_);
and
* (optionally) any expectation files (see below)
.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
@@ -125,7 +130,7 @@ input to the `wptupdate` tool.
Expectation File Format
~~~~~~~~~~~~~~~~~~~~~~~

Metadat about tests, notably including their expected results, is
Metadata about tests, notably including their expected results, is
stored in a modified ini-like format that is designed to be human
editable, but also to be machine updatable.

@@ -28,19 +28,19 @@ environment created as above::
pip install -e ./

In addition to the dependencies installed by pip, wptrunner requires
a copy of the web-platform-tests repository. That can be located
anywhere on the filesystem, but the easiest option is to put it within
the wptrunner checkout directory, as a subdirectory named ``tests``::
a copy of the web-platform-tests repository. This can be located
anywhere on the filesystem, but the easiest option is to put it
under the same parent directory as the wptrunner checkout::

git clone https://github.com/w3c/web-platform-tests.git tests
git clone https://github.com/w3c/web-platform-tests.git

It is also necessary to generate a web-platform-tests ``MANIFEST.json``
file. It's recommended to put that within the wptrunner
checkout directory, in a subdirectory named ``meta``::
file. It's recommended to also put that under the same parent directory as
the wptrunner checkout, in a directory named ``meta``::

mkdir meta
cd tests
python tools/scripts/manifest.py ../meta/MANIFEST.json
cd web-platform-tests
python manifest --path ../meta/MANIFEST.json

The ``MANIFEST.json`` file needs to be regenerated each time the
web-platform-tests checkout is updated. To aid with the update process
@@ -74,6 +74,9 @@ takes multiple options, of which the following are most significant:
``--prefs-root`` (required only when testing a Firefox binary)
The path to a directory containing Firefox test-harness preferences. [#]_

``--config`` (should default to `wptrunner.default.ini`)
The path to the config (ini) file.

.. [#] The ``--certutil-binary`` option is required when the product is
``firefox`` unless ``--ssl-type=none`` is specified.
@@ -94,10 +97,17 @@ The following examples show how to start wptrunner with various options.
Starting wptrunner
------------------

The examples below assume the following directory layout,
though no specific folder structure is required::

~/testtwf/wptrunner # wptrunner checkout
~/testtwf/web-platform-tests # web-platform-tests checkout
~/testtwf/meta # metadata

To test a Firefox Nightly build in an OS X environment, you might start
wptrunner using something similar to the following example::

wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
--binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
--certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
--prefs-root=~/mozilla-central/testing/profiles
@@ -106,7 +116,7 @@ wptrunner using something similar to the following example::
And to test a Chromium build in an OS X environment, you might start
wptrunner using something similar to the following example::

wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
--binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
--webdriver-binary=/usr/local/bin/chromedriver --product=chrome

@@ -118,7 +128,7 @@ To restrict a test run just to tests in a particular web-platform-tests
subdirectory, specify the directory name in the positional arguments after
the options; for example, run just the tests in the `dom` subdirectory::

wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
wptrunner --metadata=~/testtwf/meta --tests=~/testtwf/web-platform-tests/ \
--binary=/path/to/firefox --certutil-binary=/path/to/certutil \
--prefs-root=/path/to/testing/profiles \
dom
@@ -180,7 +190,7 @@ Configuration File

wptrunner uses a ``.ini`` file to control some configuration
sections. The file has three sections; ``[products]``,
``[paths]`` and ``[web-platform-tests]``.
``[manifest:default]`` and ``[web-platform-tests]``.

``[products]`` is used to
define the set of available products. By default this section is empty
@@ -195,12 +205,12 @@ e.g.::
chrome =
netscape4 = path/to/netscape.py

``[paths]`` specifies the default paths for the tests and metadata,
``[manifest:default]`` specifies the default paths for the tests and metadata,
relative to the config file. For example::

[paths]
tests = checkouts/web-platform-tests
metadata = /home/example/wpt/metadata
[manifest:default]
tests = ~/testtwf/web-platform-tests
metadata = ~/testtwf/meta


``[web-platform-tests]`` is used to set the properties of the upstream
@@ -192,7 +192,7 @@ def __init__(self, *args, **kwargs):

import sys, subprocess

self.device = mozdevice.ADBDevice()
self.device = mozdevice.ADBB2G()
self.device.forward("tcp:%s" % self.marionette_port,
"tcp:2828")
self.executor = None
@@ -28,7 +28,8 @@
"browser_kwargs": "browser_kwargs",
"executor_kwargs": "executor_kwargs",
"env_options": "env_options",
"run_info_extras": "run_info_extras"}
"run_info_extras": "run_info_extras",
"update_properties": "update_properties"}


def check_args(**kwargs):
@@ -54,7 +55,7 @@ def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
cache_manager, **kwargs)
executor_kwargs["close_after_done"] = True
if kwargs["timeout_multiplier"] is None:
if kwargs["gecko_e10s"] and test_type == "reftest":
if test_type == "reftest":
if run_info_data["debug"]:
executor_kwargs["timeout_multiplier"] = 4
else:
@@ -71,9 +72,14 @@ def env_options():
"certificate_domain": "web-platform.test",
"supports_debugger": True}


def run_info_extras(**kwargs):
return {"e10s": kwargs["gecko_e10s"]}


def update_properties():
return ["debug", "e10s", "os", "version", "processor", "bits"], {"debug", "e10s"}

class FirefoxBrowser(Browser):
used_ports = set()

@@ -17,7 +17,9 @@
"reftest": "ServoRefTestExecutor"},
"browser_kwargs": "browser_kwargs",
"executor_kwargs": "executor_kwargs",
"env_options": "env_options"}
"env_options": "env_options",
"run_info_extras": "run_info_extras",
"update_properties": "update_properties"}


def check_args(**kwargs):
@@ -47,8 +49,16 @@ def env_options():
"supports_debugger": True}


def run_info_extras(**kwargs):
return {"backend": kwargs["servo_backend"]}


def update_properties():
return ["debug", "os", "version", "processor", "bits", "backend"], None


def render_arg(render_backend):
return {"cpu": "--cpu"}[render_backend]
return {"cpu": "--cpu", "webrender": "--webrender"}[render_backend]


class ServoBrowser(NullBrowser):
@@ -23,7 +23,9 @@
"reftest": "ServoWebDriverRefTestExecutor"},
"browser_kwargs": "browser_kwargs",
"executor_kwargs": "executor_kwargs",
"env_options": "env_options"}
"env_options": "env_options",
"run_info_extras": "run_info_extras",
"update_properties": "update_properties"}

hosts_text = """127.0.0.1 web-platform.test
127.0.0.1 www.web-platform.test
@@ -59,6 +61,14 @@ def env_options():
"supports_debugger": True}


def run_info_extras(**kwargs):
return {"backend": kwargs["servo_backend"]}


def update_properties():
return ["debug", "os", "version", "processor", "bits", "backend"], None


def make_hosts_file():
hosts_fd, hosts_path = tempfile.mkstemp()
with os.fdopen(hosts_fd, "w") as f:
@@ -88,6 +98,7 @@ def start(self):

env = os.environ.copy()
env["HOST_FILE"] = self.hosts_path
env["RUST_BACKTRACE"] = "1"

debug_args, command = browser_command(self.binary,
[render_arg(self.render_backend), "--hard-fail",
@@ -107,12 +107,6 @@ def is_alive(self):
return True

def after_connect(self):
# Turn off debug-level logging by default since this is so verbose
with self.marionette.using_context("chrome"):
self.marionette.execute_script("""
Components.utils.import("resource://gre/modules/Log.jsm");
Log.repository.getLogger("Marionette").level = Log.Level.Info;
""")
self.load_runner("http")

def load_runner(self, protocol):
@@ -87,7 +87,7 @@ def do_test(self, test):

env = os.environ.copy()
env["HOST_FILE"] = self.hosts_path

env["RUST_BACKTRACE"] = "1"


if not self.interactive:
@@ -223,6 +223,7 @@ def screenshot(self, test, viewport_size, dpi):

env = os.environ.copy()
env["HOST_FILE"] = self.hosts_path
env["RUST_BACKTRACE"] = "1"

if not self.interactive:
self.proc = ProcessHandler(self.command,
@@ -8,6 +8,7 @@
representing the file and each subnode representing a subdirectory that should
be included or excluded.
"""
import glob
import os
import urlparse

@@ -90,29 +91,36 @@ def _add_rule(self, test_manifests, url, direction):
variant += "?" + query

maybe_path = os.path.join(rest, last)

if os.path.exists(maybe_path):
for manifest, data in test_manifests.iteritems():
rel_path = os.path.relpath(maybe_path, data["tests_path"])
if ".." not in rel_path.split(os.sep):
url = data["url_base"] + rel_path.replace(os.path.sep, "/") + variant
break
paths = glob.glob(maybe_path)

if paths:
urls = []
for path in paths:
for manifest, data in test_manifests.iteritems():
rel_path = os.path.relpath(path, data["tests_path"])
if ".." not in rel_path.split(os.sep):
urls.append(data["url_base"] + rel_path.replace(os.path.sep, "/") + variant)
break
else:
urls = [url]

assert direction in ("include", "exclude")
components = self._get_components(url)

node = self
while components:
component = components.pop()
if component not in node.child_map:
new_node = IncludeManifest(DataNode(component))
node.append(new_node)
new_node.set("skip", node.get("skip", {}))
for url in urls:
components = self._get_components(url)

node = self
while components:
component = components.pop()
if component not in node.child_map:
new_node = IncludeManifest(DataNode(component))
node.append(new_node)
new_node.set("skip", node.get("skip", {}))

node = node.child_map[component]
node = node.child_map[component]

skip = False if direction == "include" else True
node.set("skip", str(skip))
skip = False if direction == "include" else True
node.set("skip", str(skip))

def add_include(self, test_manifests, url_prefix):
"""Add a rule indicating that tests under a url path

0 comments on commit f234d99

Please sign in to comment.
You can’t perform that action at this time.