Added error_handler hook to MultiTest (#1050)

Added error_handler hook to Multitest, which allows users to define custom error handlers and add additional information to the report for better exception management.

doc/en/multitest.rst

@@ -48,73 +48,6 @@ A MultiTest instance can be constructed from the following parameters:
       def a_testcase_method(self, env, result):
         ...
 
-  In addition suites can have ``setup`` and ``teardown`` methods. The ``setup``
-  method will be executed on suite entry, prior to any testcase if present.
-  The ``teardown`` method will be executed on suite exit, after setup and all
-  ``@testcase``-decorated testcases have executed.
-
-  Again, the signature of those methods is checked at import time, and must be
-  as follows:
-
-  .. code-block:: python
-
-    def setup(self, env):
-        ...
-
-    def teardown(self, env):
-        ...
-
-  The result object can be optionally used to perform logging and basic
-  assertions:
-
-  .. code-block:: python
-
-    def setup(self, env, result):
-        ...
-
-    def teardown(self, env, result):
-        ...
-
-  To signal that either ``setup`` or ``teardown`` hasn't completed correctly,
-  you must raise an exception. Raising an exception in ``setup`` will abort
-  the execution of the testsuite, raising one in ``teardown`` will be logged
-  in the report but will not prevent the execution of the next testsuite.
-
-  Similarly suites can have ``pre_testcase`` and ``post_testcase`` methods.
-  The ``pre_testcase`` method is executed before each testcase runs, and the
-  ``post_testcase`` method is executed after each testcase finishes. Exceptions
-  raised in these methods will be logged in the report. Note that argument
-  ``name`` is populated with name of testcase.
-
-  .. code-block:: python
-
-    def pre_testcase(self, name, env, result):
-        pass
-
-    def post_testcase(self, name, env, result):
-        pass
-
-  :py:func:`@skip_if <testplan.testing.multitest.suite.skip_if>` decorator can
-  be used to annotate a testcase. It take one or more predicates, and if any of
-  them evaluated to True, then the testcase will be skipped by MultiTest instead
-  of being normally executed. The predicate's signature must name the argument
-  ``testsuite`` or a ``MethodSignatureMismatch`` exception will be raised.
-
-  .. code-block:: python
-
-    def skip_func(testsuite):
-        # It must accept an argument named "testsuite"
-        return True
-
-    @testsuite
-    class MySuite(object):
-
-        @skip_if(skip_func, lambda testsuite: False)
-        @suite.testcase
-        def case(self, env, result):
-            pass
-
-
   The :py:func:`@testcase <testplan.testing.multitest.suite.testcase>` decorated
   methods will execute in the order in which they are defined. If more than
   one suite is passed, the suites will be executed in the order in which they
@@ -150,6 +83,15 @@ A MultiTest instance can be constructed from the following parameters:
   during startup and testcases during execution. Example of initial context
   can be found :ref:`here <example_basic_initial_context>`
 
+* **Hooks**: Hooks are used to implement measures that complement the testing
+  process with necessary preparations and subsequent actions. See :ref:`example <example_best_practice>`.
+
+  - before_start: Callable to execute before starting the environment.
+  - after_start: Callable to execute after starting the environment.
+  - before_stop: Callable to execute before stopping the environment.
+  - after_stop: Callable to execute after stopping the environment.
+  - error_handler: Callable to execute when a step hits an exception.
+
 
 Example
 =======
@@ -1397,6 +1339,55 @@ Similarly, ``setup`` and ``teardown`` methods in a test suite can be limited to
 
 It's useful when ``setup`` has much initialization work that takes long, e.g. connects to a server but has no response and makes program hanging. Note that this ``@timeout`` decorator can also be used for ``pre_testcase`` and ``post_testcase``, but that is not suggested because pre/post testcase methods are called everytime before/after each testcase runs, they should be written as simple as possible.
 
+Hooks
+-----
+
+  In addition suites can have ``setup`` and ``teardown`` methods. The ``setup``
+  method will be executed on suite entry, prior to any testcase if present.
+  The ``teardown`` method will be executed on suite exit, after setup and all
+  ``@testcase``-decorated testcases have executed.
+
+  Again, the signature of those methods is checked at import time, and must be
+  as follows:
+
+  .. code-block:: python
+
+    def setup(self, env):
+        ...
+
+    def teardown(self, env):
+        ...
+
+  The result object can be optionally used to perform logging and basic
+  assertions:
+
+  .. code-block:: python
+
+    def setup(self, env, result):
+        ...
+
+    def teardown(self, env, result):
+        ...
+
+  To signal that either ``setup`` or ``teardown`` hasn't completed correctly,
+  you must raise an exception. Raising an exception in ``setup`` will abort
+  the execution of the testsuite, raising one in ``teardown`` will be logged
+  in the report but will not prevent the execution of the next testsuite.
+
+  Similarly suites can have ``pre_testcase`` and ``post_testcase`` methods.
+  The ``pre_testcase`` method is executed before each testcase runs, and the
+  ``post_testcase`` method is executed after each testcase finishes. Exceptions
+  raised in these methods will be logged in the report. Note that argument
+  ``name`` is populated with name of testcase.
+
+  .. code-block:: python
+
+    def pre_testcase(self, name, env, result):
+        pass
+
+    def post_testcase(self, name, env, result):
+        pass
+
 Xfail
 -----
 
@@ -1422,6 +1413,29 @@ If a test is expect to fail all the time, you can also use the `strict=True` the
     def fail_testcase(self, env, result):
         ...
 
+Skip if
+-------
+
+  :py:func:`@skip_if <testplan.testing.multitest.suite.skip_if>` decorator can
+  be used to annotate a testcase. It take one or more predicates, and if any of
+  them evaluated to True, then the testcase will be skipped by MultiTest instead
+  of being normally executed. The predicate's signature must name the argument
+  ``testsuite`` or a ``MethodSignatureMismatch`` exception will be raised.
+
+  .. code-block:: python
+
+    def skip_func(testsuite):
+        # It must accept an argument named "testsuite"
+        return True
+
+    @testsuite
+    class MySuite(object):
+
+        @skip_if(skip_func, lambda testsuite: False)
+        @suite.testcase
+        def case(self, env, result):
+            pass
+
 Logging
 -------
 

doc/newsfragments/1973_new.MultiTest_error_handler.rst

@@ -0,0 +1 @@
+Introducing error_handler hook for `MultiTest`, which allows users to define custom error handlers and add additional information to the report for better exception management.

examples/Best Practice/Common Utilities/test_plan.py

@@ -62,11 +62,21 @@ def after_stop_fn(env, result):
     stderr_logger(env, result)
 
     # Delete Multitest level runpath if the multitest passed.
-    # This function cleans the the runpath before the exporters
+    # This function cleans the runpath before the exporters
     # have chance collecting the files, hence commented out.
     # helper.clean_runpath_if_passed(env, result)
 
 
+def error_handler_fn(env, result):
+    # This will be executed when a step runs into an error.
+    step_results = env._environment.parent.result.step_results
+    if "run_tests" in step_results:
+        for log in step_results["run_tests"].flattened_logs:
+            if log["levelname"] == "ERROR":
+                result.log(log, description="Error log")
+    result.log("Error handler ran!")
+
+
 @test_plan(name="Example using helper")
 def main(plan):
     """
@@ -83,6 +93,7 @@ def main(plan):
             environment=[App("echo", binary="/bin/echo", args=["testplan"])],
             before_start=before_start_fn,
             after_stop=after_stop_fn,
+            error_handler=error_handler_fn,
         )
     )
 

examples/Best Practice/Common Utilities/test_plan_error_handler.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+# This plan contains tests that demonstrate failures as well.
+"""
+Example demonstrating usage of Multitest's error_handler hook.
+By using the error_handler hook users can clean-up resources and
+add additional information to the report upon unexpected Exceptions.
+"""
+
+import sys
+from testplan import test_plan
+from testplan.testing.multitest import MultiTest, testsuite, testcase
+from testplan.testing.multitest.driver.base import Driver
+from testplan.common.utils import helper
+
+
+@testsuite
+class MyTestsuite:
+    """
+    A testsuite that uses helper utilities in setup/teardown.
+    """
+
+    def setup(self, env, result):
+        # Save host environment variable in report.
+        helper.log_environment(result)
+
+        # Save current path & command line arguments in report.
+        helper.log_pwd(result)
+        helper.log_cmd(result)
+
+        # Save host hardware information in report.
+        helper.log_hardware(result)
+
+    @testcase
+    def my_testcase(self, env, result):
+        # Add your testcase here
+        result.true(True)
+
+    def teardown(self, env, result):
+        """
+        Attach testplan.log file in report.
+        """
+        helper.attach_log(result)
+
+
+def before_start_fn(env, result):
+    # Save host environment variable in report.
+    helper.log_environment(result)
+
+    # Save current path & command line arguments in report.
+    helper.log_pwd(result)
+
+
+def error_handler_fn(env, result):
+    # This will be executed when a step hits an exception.
+    step_results = env._environment.parent.result.step_results
+    if "run_tests" in step_results:
+        for log in step_results["run_tests"].flattened_logs:
+            if log["levelname"] == "ERROR":
+                result.log(log, description="Error log")
+    result.log("Error handler ran!")
+
+
+class FailingStopDriver(Driver):
+    def pre_stop(self):
+        raise Exception("Exception raised to trigger error handler hook!")
+
+
+@test_plan(name="Example running error_handler")
+def main(plan):
+    """
+    Add a MultiTest that triggers error_handler hook.
+    """
+    plan.add(
+        MultiTest(
+            name="ErrorHandlerTest",
+            suites=[
+                # This is a pre-defined testsuite that logs info to report
+                helper.TestplanExecutionInfo(),
+                MyTestsuite(),
+            ],  # shortcut: suites=[helper.TestplanExecutionInfo()]
+            environment=[FailingStopDriver("Dummy")],
+            before_start=before_start_fn,
+            error_handler=error_handler_fn,
+        )
+    )
+
+
+if __name__ == "__main__":
+    sys.exit(main().exit_code)

testplan/common/entity/base.py

@@ -953,7 +953,7 @@ def skip_step(self, step):
 
     def post_step_call(self, step):
         """
-        Callable to be invoked before each step.
+        Callable to be invoked after each step.
         """
         pass
 

testplan/testing/base.py

@@ -111,6 +111,7 @@ def get_options(cls):
             ConfigOption("after_start", default=None): start_stop_signature,
             ConfigOption("before_stop", default=None): start_stop_signature,
             ConfigOption("after_stop", default=None): start_stop_signature,
+            ConfigOption("error_handler", default=None): start_stop_signature,
             ConfigOption("test_filter"): filtering.BaseFilter,
             ConfigOption("test_sorter"): ordering.BaseSorter,
             ConfigOption("stdout_style"): test_styles.Style,
@@ -164,6 +165,7 @@ class Test(Runnable):
     :param after_start: Callable to execute after starting the environment.
     :param before_stop: Callable to execute before stopping the environment.
     :param after_stop: Callable to execute after stopping the environment.
+    :param error_handler: Callable to execute when a step hits an exception.
     :param stdout_style: Console output style.
     :param tags: User defined tag value.
     :param result: Result class definition for result object made available
@@ -191,6 +193,7 @@ def __init__(
         after_start: callable = None,
         before_stop: callable = None,
         after_stop: callable = None,
+        error_handler: callable = None,
         test_filter: filtering.BaseFilter = None,
         test_sorter: ordering.BaseSorter = None,
         stdout_style: test_styles.Style = None,
@@ -474,6 +477,7 @@ def add_post_main_steps(self) -> None:
 
     def add_post_resource_steps(self) -> None:
         """Runnable steps to run after environment stopped."""
+        self._add_step(self._run_error_handler)
         self._add_step(self.timer.end, "teardown")
 
     def run_testcases_iter(
@@ -541,6 +545,14 @@ def _get_hook_context(self, case_report):
             case_report.logged_exceptions(),
         )
 
+    def _run_error_handler(self) -> None:
+        """
+        This method runs error_handler hook.
+        """
+
+        if self.cfg.error_handler:
+            self._run_resource_hook(self.cfg.error_handler, "Error handler")
+
     def _run_resource_hook(self, hook: Callable, label: str) -> None:
         # TODO: env or env, result signature is mandatory not an "if"
         """

testplan/testing/multitest/base.py

@@ -572,9 +572,15 @@ def get_tags_index(self):
             )
         return self._tags_index
 
-    def skip_step(self, step):
+    def skip_step(self, step) -> bool:
         """Check if a step should be skipped."""
-        if step in (
+        if step == self._run_error_handler:
+            return not (
+                self.resources.start_exceptions
+                or self.resources.stop_exceptions
+                or self._get_error_logs()
+            )
+        elif step in (
             self.resources.start,
             self.resources.stop,
             self.apply_xfail_tests,
@@ -1281,6 +1287,14 @@ def _get_parent_uids(self, testsuite, testcase):
             parent_uids = [self.uid(), testsuite.uid()]
         return parent_uids
 
+    def _get_error_logs(self) -> Dict:
+        if "run_tests" in self.result.step_results:
+            return [
+                log
+                for log in self.result.step_results["run_tests"].flattened_logs
+                if log["levelname"] == "ERROR"
+            ]
+
     def _skip_testcases(self, testsuite, testcases):
         """
         Utility to forcefully skip testcases and modify their runtime status to not