Allow a port to run tests with a custom device setup

https://bugs.webkit.org/show_bug.cgi?id=160833

Reviewed by Daniel Bates.

These changes allow the IOSSimulator port to run tests in iPad mode.

This is made possible by allowing a platform to define CUSTOM_DEVICE_CLASSES,
in this case 'ipad'. When specified, any test in a directory with a suffix that matches
a custom device will be collected into a set, and run in that device's environment after
the other tests have run.

* Scripts/webkitpy/layout_tests/controllers/manager.py:
(Manager._custom_device_for_test): If the test contains a directory matching a
custom device suffix, return that custom device.
(Manager._set_up_run): Push the custom device class, if any, into options so
that the Worker can get to it.
(Manager.run): Go through the list of tests, and break it down into device-generic
tests, and tests for each device class. _run_test_subset is then called for
each collection of tests, and the results merged.
(Manager._run_test_subset): Some lines unwrapped.
(Manager._end_test_run):
(Manager._run_tests):
* Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
(SingleTestRunner.__init__): Unwrapped a line.
* Scripts/webkitpy/layout_tests/models/test_run_results.py:
(TestRunResults.merge): Add this function to merge TestRunResults
* Scripts/webkitpy/layout_tests/views/printing.py:
(Printer.print_workers_and_shards): Print the custom device, if any.
* Scripts/webkitpy/port/base.py:
(Port): Base port has empty array of custom devices.
(Port.setup_test_run): Add device_class argument.
* Scripts/webkitpy/port/driver.py:
(DriverInput.__repr__):
(Driver.check_driver.implementation):
* Scripts/webkitpy/port/efl.py:
(EflPort.setup_test_run):
* Scripts/webkitpy/port/gtk.py:
(GtkPort.setup_test_run):
* Scripts/webkitpy/port/ios.py:
(IOSSimulatorPort): Add CUSTOM_DEVICE_CLASSES for ipad.
(IOSSimulatorPort.__init__):
(IOSSimulatorPort.simulator_device_type): Use a device name from the DEVICE_CLASS_MAP
based on the custom device class.
(IOSSimulatorPort._set_device_class):
(IOSSimulatorPort._create_simulators): Factor some code into this function.
(IOSSimulatorPort.setup_test_run):
(IOSSimulatorPort.testing_device):
(IOSSimulatorPort.reset_preferences): This used to create the simulator apps, but that
seemed wrong for this function. That was moved to setup_test_run().
(IOSSimulatorPort.check_sys_deps): This function used to create testing devices,
but this happened too early, before we knew which kind of devices to create. Devices
are now created in setup_test_run().
* Scripts/webkitpy/port/win.py:
(WinPort.setup_test_run):

Canonical link: https://commits.webkit.org/178975@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@204477 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Tools/ChangeLog

@@ -1,3 +1,61 @@
+2016-08-15  Simon Fraser  <simon.fraser@apple.com>
+
+        Allow a port to run tests with a custom device setup
+        https://bugs.webkit.org/show_bug.cgi?id=160833
+
+        Reviewed by Daniel Bates.
+
+        These changes allow the IOSSimulator port to run tests in iPad mode.
+
+        This is made possible by allowing a platform to define CUSTOM_DEVICE_CLASSES,
+        in this case 'ipad'. When specified, any test in a directory with a suffix that matches
+        a custom device will be collected into a set, and run in that device's environment after
+        the other tests have run.
+
+        * Scripts/webkitpy/layout_tests/controllers/manager.py:
+        (Manager._custom_device_for_test): If the test contains a directory matching a
+        custom device suffix, return that custom device.
+        (Manager._set_up_run): Push the custom device class, if any, into options so
+        that the Worker can get to it.
+        (Manager.run): Go through the list of tests, and break it down into device-generic
+        tests, and tests for each device class. _run_test_subset is then called for
+        each collection of tests, and the results merged.
+        (Manager._run_test_subset): Some lines unwrapped.
+        (Manager._end_test_run):
+        (Manager._run_tests):
+        * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
+        (SingleTestRunner.__init__): Unwrapped a line.
+        * Scripts/webkitpy/layout_tests/models/test_run_results.py:
+        (TestRunResults.merge): Add this function to merge TestRunResults
+        * Scripts/webkitpy/layout_tests/views/printing.py:
+        (Printer.print_workers_and_shards): Print the custom device, if any.
+        * Scripts/webkitpy/port/base.py:
+        (Port): Base port has empty array of custom devices.
+        (Port.setup_test_run): Add device_class argument.
+        * Scripts/webkitpy/port/driver.py:
+        (DriverInput.__repr__):
+        (Driver.check_driver.implementation):
+        * Scripts/webkitpy/port/efl.py:
+        (EflPort.setup_test_run):
+        * Scripts/webkitpy/port/gtk.py:
+        (GtkPort.setup_test_run):
+        * Scripts/webkitpy/port/ios.py:
+        (IOSSimulatorPort): Add CUSTOM_DEVICE_CLASSES for ipad.
+        (IOSSimulatorPort.__init__):
+        (IOSSimulatorPort.simulator_device_type): Use a device name from the DEVICE_CLASS_MAP
+        based on the custom device class.
+        (IOSSimulatorPort._set_device_class):
+        (IOSSimulatorPort._create_simulators): Factor some code into this function.
+        (IOSSimulatorPort.setup_test_run):
+        (IOSSimulatorPort.testing_device):
+        (IOSSimulatorPort.reset_preferences): This used to create the simulator apps, but that
+        seemed wrong for this function. That was moved to setup_test_run().
+        (IOSSimulatorPort.check_sys_deps): This function used to create testing devices,
+        but this happened too early, before we knew which kind of devices to create. Devices
+        are now created in setup_test_run().
+        * Scripts/webkitpy/port/win.py:
+        (WinPort.setup_test_run):
+
 2016-08-15  Daniel Bates  <dabates@apple.com>
 
         Cannot build WebKit for iOS device using Xcode 7.3/iOS 9.3 public SDK due to missing

Tools/Scripts/webkitpy/layout_tests/controllers/manager.py

@@ -39,6 +39,7 @@
 import random
 import sys
 import time
+from collections import defaultdict
 
 from webkitpy.common.checkout.scm.detection import SCMDetector
 from webkitpy.common.net.file_uploader import FileUploader
@@ -97,6 +98,13 @@ def _is_websocket_test(self, test):
     def _is_web_platform_test(self, test):
         return self.web_platform_test_subdir in test
 
+    def _custom_device_for_test(self, test):
+        for device_class in self._port.CUSTOM_DEVICE_CLASSES:
+            directory_suffix = device_class + self._port.TEST_PATH_SEPARATOR
+            if directory_suffix in test:
+                return device_class
+        return None
+
     def _http_tests(self, test_names):
         return set(test for test in test_names if self._is_http_test(test))
 
@@ -141,12 +149,14 @@ def _update_worker_count(self, test_names):
         worker_count = self._runner.get_worker_count(test_inputs, int(self._options.child_processes))
         self._options.child_processes = worker_count
 
-    def _set_up_run(self, test_names):
+    def _set_up_run(self, test_names, device_class=None):
         self._printer.write_update("Checking build ...")
         if not self._port.check_build(self.needs_servers(test_names)):
             _log.error("Build check failed")
             return False
 
+        self._options.device_class = device_class
+
         # This must be started before we check the system dependencies,
         # since the helper may do things to make the setup correct.
         self._printer.write_update("Starting helper ...")
@@ -169,7 +179,7 @@ def _set_up_run(self, test_names):
         # Create the output directory if it doesn't already exist.
         self._port.host.filesystem.maybe_make_directory(self._results_directory)
 
-        self._port.setup_test_run()
+        self._port.setup_test_run(self._options.device_class)
         return True
 
     def run(self, args):
@@ -194,13 +204,56 @@ def run(self, args):
             _log.critical('No tests to run.')
             return test_run_results.RunDetails(exit_code=-1)
 
-        try:
+        default_device_tests = []
+
+        # Look for tests with custom device requirements.
+        custom_device_tests = defaultdict(list)
+        for test_file in tests_to_run:
+            custom_device = self._custom_device_for_test(test_file)
+            if custom_device:
+                custom_device_tests[custom_device].append(test_file)
+            else:
+                default_device_tests.append(test_file)
+
+        if custom_device_tests:
+            for device_class in custom_device_tests:
+                _log.debug('{} tests use device {}'.format(len(custom_device_tests[device_class]), device_class))
+
+        initial_results = None
+        retry_results = None
+        enabled_pixel_tests_in_retry = False
+
+        if default_device_tests:
+            _log.info('')
+            _log.info("Running %s", pluralize(len(tests_to_run), "test"))
+            _log.info('')
             if not self._set_up_run(tests_to_run):
                 return test_run_results.RunDetails(exit_code=-1)
 
+            initial_results, retry_results, enabled_pixel_tests_in_retry = self._run_test_subset(default_device_tests, tests_to_skip)
+
+        for device_class in custom_device_tests:
+            device_tests = custom_device_tests[device_class]
+            if device_tests:
+                _log.info('')
+                _log.info('Running %s for %s', pluralize(len(device_tests), "test"), device_class)
+                _log.info('')
+                if not self._set_up_run(device_tests, device_class):
+                    return test_run_results.RunDetails(exit_code=-1)
+
+                device_initial_results, device_retry_results, device_enabled_pixel_tests_in_retry = self._run_test_subset(device_tests, tests_to_skip)
+
+                initial_results = initial_results.merge(device_initial_results) if initial_results else device_initial_results
+                retry_results = retry_results.merge(device_retry_results) if retry_results else device_retry_results
+                enabled_pixel_tests_in_retry |= device_enabled_pixel_tests_in_retry
+
+        end_time = time.time()
+        return self._end_test_run(start_time, end_time, initial_results, retry_results, enabled_pixel_tests_in_retry)
+
+    def _run_test_subset(self, tests_to_run, tests_to_skip):
+        try:
             enabled_pixel_tests_in_retry = False
-            initial_results = self._run_tests(tests_to_run, tests_to_skip, self._options.repeat_each, self._options.iterations,
-                int(self._options.child_processes), retrying=False)
+            initial_results = self._run_tests(tests_to_run, tests_to_skip, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False)
 
             tests_to_retry = self._tests_to_retry(initial_results, include_crashes=self._port.should_retry_crashes())
             # Don't retry failures when interrupted by user or failures limit exception.
@@ -211,8 +264,7 @@ def run(self, args):
                 _log.info('')
                 _log.info("Retrying %s ..." % pluralize(len(tests_to_retry), "unexpected failure"))
                 _log.info('')
-                retry_results = self._run_tests(tests_to_retry, tests_to_skip=set(), repeat_each=1, iterations=1,
-                    num_workers=1, retrying=True)
+                retry_results = self._run_tests(tests_to_retry, tests_to_skip=set(), repeat_each=1, iterations=1, num_workers=1, retrying=True)
 
                 if enabled_pixel_tests_in_retry:
                     self._options.pixel_tests = False
@@ -221,10 +273,12 @@ def run(self, args):
         finally:
             self._clean_up_run()
 
-        end_time = time.time()
+        return (initial_results, retry_results, enabled_pixel_tests_in_retry)
 
+    def _end_test_run(self, start_time, end_time, initial_results, retry_results, enabled_pixel_tests_in_retry):
         # Some crash logs can take a long time to be written out so look
         # for new logs after the test run finishes.
+
         _log.debug("looking for new crash logs")
         self._look_for_new_crash_logs(initial_results, start_time)
         if retry_results:
@@ -259,6 +313,7 @@ def _run_tests(self, tests_to_run, tests_to_skip, repeat_each, iterations, num_w
         needs_websockets = any(self._is_websocket_test(test) for test in tests_to_run)
 
         test_inputs = self._get_test_inputs(tests_to_run, repeat_each, iterations)
+
         return self._runner.run_tests(self._expectations, test_inputs, tests_to_skip, num_workers, needs_http, needs_websockets, needs_web_platform_test_server, retrying)
 
     def _clean_up_run(self):

Tools/Scripts/webkitpy/layout_tests/controllers/manager_unittest.py

@@ -37,6 +37,7 @@
 from webkitpy.layout_tests.controllers.manager import Manager
 from webkitpy.layout_tests.models import test_expectations
 from webkitpy.layout_tests.models.test_run_results import TestRunResults
+from webkitpy.port.test import TestPort
 from webkitpy.thirdparty.mock import Mock
 from webkitpy.tool.mocktool import MockOptions
 
@@ -99,3 +100,19 @@ def get_manager():
         run_results = TestRunResults(expectations, len(tests))
         manager = get_manager()
         manager._look_for_new_crash_logs(run_results, time.time())
+
+    def test_uses_custom_device(self):
+        class MockCustomDevicePort(TestPort):
+            CUSTOM_DEVICE_CLASSES = ['starship']            
+            def __init__(self, host):
+                super(MockCustomDevicePort, self).__init__(host)
+
+        def get_manager():
+            host = MockHost()
+            port = MockCustomDevicePort(host)
+            manager = Manager(port, options=MockOptions(test_list=['fast/test-starship/lasers.html'], http=True), printer=Mock())
+            return manager
+
+        manager = get_manager()
+        self.assertTrue(manager._custom_device_for_test('fast/test-starship/lasers.html') == 'starship')
+

Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py

@@ -70,8 +70,7 @@ def __init__(self, port, options, results_directory, worker_name, driver, test_i
             for suffix in ('.txt', '.png', '.wav'):
                 expected_filename = self._port.expected_filename(self._test_name, suffix)
                 if self._filesystem.exists(expected_filename):
-                    _log.error('%s is a reftest, but has an unused expectation file. Please remove %s.',
-                        self._test_name, expected_filename)
+                    _log.error('%s is a reftest, but has an unused expectation file. Please remove %s.', self._test_name, expected_filename)
 
     def _expected_driver_output(self):
         return DriverOutput(self._port.expected_text(self._test_name),

Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py

@@ -93,6 +93,30 @@ def add(self, test_result, expected, test_is_slow):
         if test_is_slow:
             self.slow_tests.add(test_result.test_name)
 
+    def merge(self, test_run_results):
+        # self.expectations should be the same for both
+        self.total += test_run_results.total
+        self.remaining += test_run_results.remaining
+        self.expected += test_run_results.expected
+        self.unexpected += test_run_results.unexpected
+        self.unexpected_failures += test_run_results.unexpected_failures
+        self.unexpected_crashes += test_run_results.unexpected_crashes
+        self.unexpected_timeouts += test_run_results.unexpected_timeouts
+        self.tests_by_expectation.update(test_run_results.tests_by_expectation)
+        self.tests_by_timeline.update(test_run_results.tests_by_timeline)
+        self.results_by_name.update(test_run_results.results_by_name)
+        self.all_results += test_run_results.all_results
+        self.unexpected_results_by_name.update(test_run_results.unexpected_results_by_name)
+        self.failures_by_name.update(test_run_results.failures_by_name)
+        self.total_failures += test_run_results.total_failures
+        self.expected_skips += test_run_results.expected_skips
+        self.tests_by_expectation.update(test_run_results.tests_by_expectation)
+        self.tests_by_timeline.update(test_run_results.tests_by_timeline)
+        self.slow_tests.update(test_run_results.slow_tests)
+
+        self.interrupted |= test_run_results.interrupted
+        self.keyboard_interrupted |= test_run_results.keyboard_interrupted
+        return self
 
 class RunDetails(object):
     def __init__(self, exit_code, summarized_results=None, initial_results=None, retry_results=None, enabled_pixel_tests_in_retry=False):

Tools/Scripts/webkitpy/layout_tests/views/printing.py

@@ -112,12 +112,14 @@ def print_expected(self, run_results, tests_with_result_type_callback):
 
     def print_workers_and_shards(self, num_workers, num_shards):
         driver_name = self._port.driver_name()
+
+        device_suffix = ' for device "{}"'.format(self._options.device_class) if self._options.device_class else ''
         if num_workers == 1:
-            self._print_default("Running 1 %s." % driver_name)
-            self._print_debug("(%s)." % grammar.pluralize(num_shards, "shard"))
+            self._print_default('Running 1 {}{}.'.format(driver_name, device_suffix))
+            self._print_debug('({}).'.format(grammar.pluralize(num_shards, "shard")))
         else:
-            self._print_default("Running %s in parallel." % (grammar.pluralize(num_workers, driver_name)))
-            self._print_debug("(%d shards)." % num_shards)
+            self._print_default('Running {} in parallel{}.'.format(grammar.pluralize(num_workers, driver_name), device_suffix))
+            self._print_debug('({} shards).'.format(num_shards))
         self._print_default('')
 
     def _print_expected_results_of_type(self, run_results, result_type, result_type_str, tests_with_result_type_callback):

Tools/Scripts/webkitpy/port/base.py

@@ -82,6 +82,8 @@ class Port(object):
 
     DEFAULT_ARCHITECTURE = 'x86'
 
+    CUSTOM_DEVICE_CLASSES = []
+
     @classmethod
     def determine_full_port_name(cls, host, options, port_name):
         """Return a fully-specified port name that can be used to construct objects."""
@@ -803,7 +805,7 @@ def default_results_directory(self):
         # to have multiple copies of webkit checked out and built.
         return self._build_path('layout-test-results')
 
-    def setup_test_run(self):
+    def setup_test_run(self, device_class=None):
         """Perform port-specific work at the beginning of a test run."""
         pass
 

Tools/Scripts/webkitpy/port/driver.py

@@ -52,6 +52,9 @@ def __init__(self, test_name, timeout, image_hash, should_run_pixel_test, args=N
         self.should_run_pixel_test = should_run_pixel_test
         self.args = args or []
 
+    def __repr__(self):
+        return "DriverInput(test_name='{}', timeout={}, image_hash={}, should_run_pixel_test={}'".format(self.test_name, self.timeout, self.image_hash, self.should_run_pixel_test)
+
 
 class DriverOutput(object):
     """Groups information about a output from driver for easy passing
@@ -587,6 +590,7 @@ def check_driver(port):
         return True
 
 
+# FIXME: this should be abstracted out via the Port subclass somehow.
 class IOSSimulatorDriver(Driver):
     def cmd_line(self, pixel_tests, per_test_args):
         cmd = super(IOSSimulatorDriver, self).cmd_line(pixel_tests, per_test_args)

Tools/Scripts/webkitpy/port/efl.py

@@ -54,8 +54,8 @@ def __init__(self, *args, **kwargs):
     def _port_flag_for_scripts(self):
         return "--efl"
 
-    def setup_test_run(self):
-        super(EflPort, self).setup_test_run()
+    def setup_test_run(self, device_class=None):
+        super(EflPort, self).setup_test_run(device_class)
         self._pulseaudio_sanitizer.unload_pulseaudio_module()
 
     def setup_environ_for_server(self, server_name=None):

Tools/Scripts/webkitpy/port/gtk.py

@@ -100,8 +100,8 @@ def driver_stop_timeout(self):
             return self.default_timeout_ms()
         return super(GtkPort, self).driver_stop_timeout()
 
-    def setup_test_run(self):
-        super(GtkPort, self).setup_test_run()
+    def setup_test_run(self, device_class=None):
+        super(GtkPort, self).setup_test_run(device_class)
         self._pulseaudio_sanitizer.unload_pulseaudio_module()
 
         if self.get_option("leaks"):