Skip to content
Permalink
Browse files
Allow a port to run tests with a custom device setup
https://bugs.webkit.org/show_bug.cgi?id=160833

Reviewed by Daniel Bates.

These changes allow the IOSSimulator port to run tests in iPad mode.

This is made possible by allowing a platform to define CUSTOM_DEVICE_CLASSES,
in this case 'ipad'. When specified, any test in a directory with a suffix that matches
a custom device will be collected into a set, and run in that device's environment after
the other tests have run.

* Scripts/webkitpy/layout_tests/controllers/manager.py:
(Manager._custom_device_for_test): If the test contains a directory matching a
custom device suffix, return that custom device.
(Manager._set_up_run): Push the custom device class, if any, into options so
that the Worker can get to it.
(Manager.run): Go through the list of tests, and break it down into device-generic
tests, and tests for each device class. _run_test_subset is then called for
each collection of tests, and the results merged.
(Manager._run_test_subset): Some lines unwrapped.
(Manager._end_test_run):
(Manager._run_tests):
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner.__init__): Unwrapped a line.
* Scripts/webkitpy/layout_tests/models/test_run_results.py:
(TestRunResults.merge): Add this function to merge TestRunResults
* Scripts/webkitpy/layout_tests/views/printing.py:
(Printer.print_workers_and_shards): Print the custom device, if any.
* Scripts/webkitpy/port/base.py:
(Port): Base port has empty array of custom devices.
(Port.setup_test_run): Add device_class argument.
* Scripts/webkitpy/port/driver.py:
(DriverInput.__repr__):
(Driver.check_driver.implementation):
* Scripts/webkitpy/port/efl.py:
(EflPort.setup_test_run):
* Scripts/webkitpy/port/gtk.py:
(GtkPort.setup_test_run):
* Scripts/webkitpy/port/ios.py:
(IOSSimulatorPort): Add CUSTOM_DEVICE_CLASSES for ipad.
(IOSSimulatorPort.__init__):
(IOSSimulatorPort.simulator_device_type): Use a device name from the DEVICE_CLASS_MAP
based on the custom device class.
(IOSSimulatorPort._set_device_class):
(IOSSimulatorPort._create_simulators): Factor some code into this function.
(IOSSimulatorPort.setup_test_run):
(IOSSimulatorPort.testing_device):
(IOSSimulatorPort.reset_preferences): This used to create the simulator apps, but that
seemed wrong for this function. That was moved to setup_test_run().
(IOSSimulatorPort.check_sys_deps): This function used to create testing devices,
but this happened too early, before we knew which kind of devices to create. Devices
are now created in setup_test_run().
* Scripts/webkitpy/port/win.py:
(WinPort.setup_test_run):

Canonical link: https://commits.webkit.org/178975@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@204477 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
smfr committed Aug 15, 2016
1 parent d680f95 commit fb40807d5b6d7ac05607f277449bc6e5464e9379
Show file tree
Hide file tree
Showing 13 changed files with 240 additions and 45 deletions.
@@ -1,3 +1,61 @@
2016-08-15 Simon Fraser <simon.fraser@apple.com>

Allow a port to run tests with a custom device setup
https://bugs.webkit.org/show_bug.cgi?id=160833

Reviewed by Daniel Bates.

These changes allow the IOSSimulator port to run tests in iPad mode.

This is made possible by allowing a platform to define CUSTOM_DEVICE_CLASSES,
in this case 'ipad'. When specified, any test in a directory with a suffix that matches
a custom device will be collected into a set, and run in that device's environment after
the other tests have run.

* Scripts/webkitpy/layout_tests/controllers/manager.py:
(Manager._custom_device_for_test): If the test contains a directory matching a
custom device suffix, return that custom device.
(Manager._set_up_run): Push the custom device class, if any, into options so
that the Worker can get to it.
(Manager.run): Go through the list of tests, and break it down into device-generic
tests, and tests for each device class. _run_test_subset is then called for
each collection of tests, and the results merged.
(Manager._run_test_subset): Some lines unwrapped.
(Manager._end_test_run):
(Manager._run_tests):
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner.__init__): Unwrapped a line.
* Scripts/webkitpy/layout_tests/models/test_run_results.py:
(TestRunResults.merge): Add this function to merge TestRunResults
* Scripts/webkitpy/layout_tests/views/printing.py:
(Printer.print_workers_and_shards): Print the custom device, if any.
* Scripts/webkitpy/port/base.py:
(Port): Base port has empty array of custom devices.
(Port.setup_test_run): Add device_class argument.
* Scripts/webkitpy/port/driver.py:
(DriverInput.__repr__):
(Driver.check_driver.implementation):
* Scripts/webkitpy/port/efl.py:
(EflPort.setup_test_run):
* Scripts/webkitpy/port/gtk.py:
(GtkPort.setup_test_run):
* Scripts/webkitpy/port/ios.py:
(IOSSimulatorPort): Add CUSTOM_DEVICE_CLASSES for ipad.
(IOSSimulatorPort.__init__):
(IOSSimulatorPort.simulator_device_type): Use a device name from the DEVICE_CLASS_MAP
based on the custom device class.
(IOSSimulatorPort._set_device_class):
(IOSSimulatorPort._create_simulators): Factor some code into this function.
(IOSSimulatorPort.setup_test_run):
(IOSSimulatorPort.testing_device):
(IOSSimulatorPort.reset_preferences): This used to create the simulator apps, but that
seemed wrong for this function. That was moved to setup_test_run().
(IOSSimulatorPort.check_sys_deps): This function used to create testing devices,
but this happened too early, before we knew which kind of devices to create. Devices
are now created in setup_test_run().
* Scripts/webkitpy/port/win.py:
(WinPort.setup_test_run):

2016-08-15 Daniel Bates <dabates@apple.com>

Cannot build WebKit for iOS device using Xcode 7.3/iOS 9.3 public SDK due to missing
@@ -39,6 +39,7 @@
import random
import sys
import time
from collections import defaultdict

from webkitpy.common.checkout.scm.detection import SCMDetector
from webkitpy.common.net.file_uploader import FileUploader
@@ -97,6 +98,13 @@ def _is_websocket_test(self, test):
def _is_web_platform_test(self, test):
return self.web_platform_test_subdir in test

def _custom_device_for_test(self, test):
for device_class in self._port.CUSTOM_DEVICE_CLASSES:
directory_suffix = device_class + self._port.TEST_PATH_SEPARATOR
if directory_suffix in test:
return device_class
return None

def _http_tests(self, test_names):
return set(test for test in test_names if self._is_http_test(test))

@@ -141,12 +149,14 @@ def _update_worker_count(self, test_names):
worker_count = self._runner.get_worker_count(test_inputs, int(self._options.child_processes))
self._options.child_processes = worker_count

def _set_up_run(self, test_names):
def _set_up_run(self, test_names, device_class=None):
self._printer.write_update("Checking build ...")
if not self._port.check_build(self.needs_servers(test_names)):
_log.error("Build check failed")
return False

self._options.device_class = device_class

# This must be started before we check the system dependencies,
# since the helper may do things to make the setup correct.
self._printer.write_update("Starting helper ...")
@@ -169,7 +179,7 @@ def _set_up_run(self, test_names):
# Create the output directory if it doesn't already exist.
self._port.host.filesystem.maybe_make_directory(self._results_directory)

self._port.setup_test_run()
self._port.setup_test_run(self._options.device_class)
return True

def run(self, args):
@@ -194,13 +204,56 @@ def run(self, args):
_log.critical('No tests to run.')
return test_run_results.RunDetails(exit_code=-1)

try:
default_device_tests = []

# Look for tests with custom device requirements.
custom_device_tests = defaultdict(list)
for test_file in tests_to_run:
custom_device = self._custom_device_for_test(test_file)
if custom_device:
custom_device_tests[custom_device].append(test_file)
else:
default_device_tests.append(test_file)

if custom_device_tests:
for device_class in custom_device_tests:
_log.debug('{} tests use device {}'.format(len(custom_device_tests[device_class]), device_class))

initial_results = None
retry_results = None
enabled_pixel_tests_in_retry = False

if default_device_tests:
_log.info('')
_log.info("Running %s", pluralize(len(tests_to_run), "test"))
_log.info('')
if not self._set_up_run(tests_to_run):
return test_run_results.RunDetails(exit_code=-1)

initial_results, retry_results, enabled_pixel_tests_in_retry = self._run_test_subset(default_device_tests, tests_to_skip)

for device_class in custom_device_tests:
device_tests = custom_device_tests[device_class]
if device_tests:
_log.info('')
_log.info('Running %s for %s', pluralize(len(device_tests), "test"), device_class)
_log.info('')
if not self._set_up_run(device_tests, device_class):
return test_run_results.RunDetails(exit_code=-1)

device_initial_results, device_retry_results, device_enabled_pixel_tests_in_retry = self._run_test_subset(device_tests, tests_to_skip)

initial_results = initial_results.merge(device_initial_results) if initial_results else device_initial_results
retry_results = retry_results.merge(device_retry_results) if retry_results else device_retry_results
enabled_pixel_tests_in_retry |= device_enabled_pixel_tests_in_retry

end_time = time.time()
return self._end_test_run(start_time, end_time, initial_results, retry_results, enabled_pixel_tests_in_retry)

def _run_test_subset(self, tests_to_run, tests_to_skip):
try:
enabled_pixel_tests_in_retry = False
initial_results = self._run_tests(tests_to_run, tests_to_skip, self._options.repeat_each, self._options.iterations,
int(self._options.child_processes), retrying=False)
initial_results = self._run_tests(tests_to_run, tests_to_skip, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False)

tests_to_retry = self._tests_to_retry(initial_results, include_crashes=self._port.should_retry_crashes())
# Don't retry failures when interrupted by user or failures limit exception.
@@ -211,8 +264,7 @@ def run(self, args):
_log.info('')
_log.info("Retrying %s ..." % pluralize(len(tests_to_retry), "unexpected failure"))
_log.info('')
retry_results = self._run_tests(tests_to_retry, tests_to_skip=set(), repeat_each=1, iterations=1,
num_workers=1, retrying=True)
retry_results = self._run_tests(tests_to_retry, tests_to_skip=set(), repeat_each=1, iterations=1, num_workers=1, retrying=True)

if enabled_pixel_tests_in_retry:
self._options.pixel_tests = False
@@ -221,10 +273,12 @@ def run(self, args):
finally:
self._clean_up_run()

end_time = time.time()
return (initial_results, retry_results, enabled_pixel_tests_in_retry)

def _end_test_run(self, start_time, end_time, initial_results, retry_results, enabled_pixel_tests_in_retry):
# Some crash logs can take a long time to be written out so look
# for new logs after the test run finishes.

_log.debug("looking for new crash logs")
self._look_for_new_crash_logs(initial_results, start_time)
if retry_results:
@@ -259,6 +313,7 @@ def _run_tests(self, tests_to_run, tests_to_skip, repeat_each, iterations, num_w
needs_websockets = any(self._is_websocket_test(test) for test in tests_to_run)

test_inputs = self._get_test_inputs(tests_to_run, repeat_each, iterations)

return self._runner.run_tests(self._expectations, test_inputs, tests_to_skip, num_workers, needs_http, needs_websockets, needs_web_platform_test_server, retrying)

def _clean_up_run(self):
@@ -37,6 +37,7 @@
from webkitpy.layout_tests.controllers.manager import Manager
from webkitpy.layout_tests.models import test_expectations
from webkitpy.layout_tests.models.test_run_results import TestRunResults
from webkitpy.port.test import TestPort
from webkitpy.thirdparty.mock import Mock
from webkitpy.tool.mocktool import MockOptions

@@ -99,3 +100,19 @@ def get_manager():
run_results = TestRunResults(expectations, len(tests))
manager = get_manager()
manager._look_for_new_crash_logs(run_results, time.time())

def test_uses_custom_device(self):
class MockCustomDevicePort(TestPort):
CUSTOM_DEVICE_CLASSES = ['starship']
def __init__(self, host):
super(MockCustomDevicePort, self).__init__(host)

def get_manager():
host = MockHost()
port = MockCustomDevicePort(host)
manager = Manager(port, options=MockOptions(test_list=['fast/test-starship/lasers.html'], http=True), printer=Mock())
return manager

manager = get_manager()
self.assertTrue(manager._custom_device_for_test('fast/test-starship/lasers.html') == 'starship')

@@ -70,8 +70,7 @@ def __init__(self, port, options, results_directory, worker_name, driver, test_i
for suffix in ('.txt', '.png', '.wav'):
expected_filename = self._port.expected_filename(self._test_name, suffix)
if self._filesystem.exists(expected_filename):
_log.error('%s is a reftest, but has an unused expectation file. Please remove %s.',
self._test_name, expected_filename)
_log.error('%s is a reftest, but has an unused expectation file. Please remove %s.', self._test_name, expected_filename)

def _expected_driver_output(self):
return DriverOutput(self._port.expected_text(self._test_name),
@@ -93,6 +93,30 @@ def add(self, test_result, expected, test_is_slow):
if test_is_slow:
self.slow_tests.add(test_result.test_name)

def merge(self, test_run_results):
# self.expectations should be the same for both
self.total += test_run_results.total
self.remaining += test_run_results.remaining
self.expected += test_run_results.expected
self.unexpected += test_run_results.unexpected
self.unexpected_failures += test_run_results.unexpected_failures
self.unexpected_crashes += test_run_results.unexpected_crashes
self.unexpected_timeouts += test_run_results.unexpected_timeouts
self.tests_by_expectation.update(test_run_results.tests_by_expectation)
self.tests_by_timeline.update(test_run_results.tests_by_timeline)
self.results_by_name.update(test_run_results.results_by_name)
self.all_results += test_run_results.all_results
self.unexpected_results_by_name.update(test_run_results.unexpected_results_by_name)
self.failures_by_name.update(test_run_results.failures_by_name)
self.total_failures += test_run_results.total_failures
self.expected_skips += test_run_results.expected_skips
self.tests_by_expectation.update(test_run_results.tests_by_expectation)
self.tests_by_timeline.update(test_run_results.tests_by_timeline)
self.slow_tests.update(test_run_results.slow_tests)

self.interrupted |= test_run_results.interrupted
self.keyboard_interrupted |= test_run_results.keyboard_interrupted
return self

class RunDetails(object):
def __init__(self, exit_code, summarized_results=None, initial_results=None, retry_results=None, enabled_pixel_tests_in_retry=False):
@@ -112,12 +112,14 @@ def print_expected(self, run_results, tests_with_result_type_callback):

def print_workers_and_shards(self, num_workers, num_shards):
driver_name = self._port.driver_name()

device_suffix = ' for device "{}"'.format(self._options.device_class) if self._options.device_class else ''
if num_workers == 1:
self._print_default("Running 1 %s." % driver_name)
self._print_debug("(%s)." % grammar.pluralize(num_shards, "shard"))
self._print_default('Running 1 {}{}.'.format(driver_name, device_suffix))
self._print_debug('({}).'.format(grammar.pluralize(num_shards, "shard")))
else:
self._print_default("Running %s in parallel." % (grammar.pluralize(num_workers, driver_name)))
self._print_debug("(%d shards)." % num_shards)
self._print_default('Running {} in parallel{}.'.format(grammar.pluralize(num_workers, driver_name), device_suffix))
self._print_debug('({} shards).'.format(num_shards))
self._print_default('')

def _print_expected_results_of_type(self, run_results, result_type, result_type_str, tests_with_result_type_callback):
@@ -82,6 +82,8 @@ class Port(object):

DEFAULT_ARCHITECTURE = 'x86'

CUSTOM_DEVICE_CLASSES = []

@classmethod
def determine_full_port_name(cls, host, options, port_name):
"""Return a fully-specified port name that can be used to construct objects."""
@@ -803,7 +805,7 @@ def default_results_directory(self):
# to have multiple copies of webkit checked out and built.
return self._build_path('layout-test-results')

def setup_test_run(self):
def setup_test_run(self, device_class=None):
"""Perform port-specific work at the beginning of a test run."""
pass

@@ -52,6 +52,9 @@ def __init__(self, test_name, timeout, image_hash, should_run_pixel_test, args=N
self.should_run_pixel_test = should_run_pixel_test
self.args = args or []

def __repr__(self):
return "DriverInput(test_name='{}', timeout={}, image_hash={}, should_run_pixel_test={}'".format(self.test_name, self.timeout, self.image_hash, self.should_run_pixel_test)


class DriverOutput(object):
"""Groups information about a output from driver for easy passing
@@ -587,6 +590,7 @@ def check_driver(port):
return True


# FIXME: this should be abstracted out via the Port subclass somehow.
class IOSSimulatorDriver(Driver):
def cmd_line(self, pixel_tests, per_test_args):
cmd = super(IOSSimulatorDriver, self).cmd_line(pixel_tests, per_test_args)
@@ -54,8 +54,8 @@ def __init__(self, *args, **kwargs):
def _port_flag_for_scripts(self):
return "--efl"

def setup_test_run(self):
super(EflPort, self).setup_test_run()
def setup_test_run(self, device_class=None):
super(EflPort, self).setup_test_run(device_class)
self._pulseaudio_sanitizer.unload_pulseaudio_module()

def setup_environ_for_server(self, server_name=None):
@@ -100,8 +100,8 @@ def driver_stop_timeout(self):
return self.default_timeout_ms()
return super(GtkPort, self).driver_stop_timeout()

def setup_test_run(self):
super(GtkPort, self).setup_test_run()
def setup_test_run(self, device_class=None):
super(GtkPort, self).setup_test_run(device_class)
self._pulseaudio_sanitizer.unload_pulseaudio_module()

if self.get_option("leaks"):

0 comments on commit fb40807

Please sign in to comment.