Merge pull request #110 from gregoil/fix_order_n_update_doc

Fix order & update doc

docs/advanced/custom_output_handlers.rst

@@ -19,12 +19,27 @@ You can make your own Output Handler, following the next two steps:
   :class:`rotest.core.result.handlers.abstract_handler.AbstractResultHandler`,
   and overriding the relevant methods.
 
-* Register the above inheriting class as an entrypoint, in a setup.py file, and
-  make sure it's being install on the environment.
+* Register the above inheriting class as an entrypoint in your setup.py file inside ``setup()``:
 
-For an example, please refer to
+    .. code-block:: python
+
+        entry_points={
+            "rotest.result_handlers":
+                ["<handler tag, e.g. my_handler> = <import path to the monitor's module>:<monitor class name>"]
+        },
+
+* Make sure it's being installed in the environment by calling
+
+    .. code-block:: console
+
+        python setup.py develop
+
+
+For an example, you can refer to
 `rotest_reportportal <https://github.com/gregoil/rotest_reportportal>`_ plugin.
 
+.. _available_events:
+
 Available Events
 ================
 

docs/advanced/monitors.rst

@@ -0,0 +1,84 @@
+=============
+Test Monitors
+=============
+
+Purpose
+=======
+
+Monitors are custom output handlers that are meant to give further validation
+of tests in runtime, or save extra information about the tests.
+
+The features of monitors:
+
+* They can be applied or dropped as easily as adding a new output handler to the list.
+
+* They enable extending sets of tests with additional validations without altering their code.
+
+* They can run in the background (in another thread).
+
+A classic example is monitoring CPU usage during tests, or a resource's log file.
+
+Writing A Monitor
+=================
+
+There are two monitor classes which you can inherit from:
+
+.. autoclass:: rotest.core.result.monitor.monitor.AbstractMonitor
+    :members: SINGLE_FAILURE, CYCLE, run_monitor, fail_test
+
+
+.. autoclass:: rotest.core.result.monitor.monitor.AbstractResourceMonitor
+    :members: RESOURCE_NAME
+
+
+There are two types of monitors:
+
+* Monitors that only react to test events, e.g. taking a screen-shot on error.
+
+  Since monitors inherit from ``AbstractResultHandler``, you can react
+  to any test event by overriding the appropriate method.
+
+  See :ref:`available_events` for a list of events.
+
+  Each of those event methods gets the test instance as the first parameter,
+  through which you can access its fields (``test.<resource>``, ``test.config``,
+  ``test.work_dir``, etc.)
+
+* Monitors that run in the background and periodically save data or run a validation,
+  like the above suggested CPU usage monitor.
+
+  To create such a monitor, simply override the class field ``CYCLE`` and the
+  method ``run_monitor``.
+
+  Again, the ``run_monitor`` method (which is called periodically after `setUp`
+  and until `tearDown`) gets the test instance as a parameter, through which
+  you can get what you need.
+
+  Note that the monitor thread is created only for upper tests, i.e. ``TestCases``
+  or topmost ``TestFlows``.
+
+  Remember that you might need to use some synchronization mechanism since
+  you're running in a different thread yet using the test's own resources.
+
+
+Use the method ``fail_test`` to add monitor failures to your tests in the background, e.g.
+
+    .. code-block:: python
+
+        self.fail_test(test, "Reached 100% CPU usage")
+
+
+Note that when using ``TestBlocks`` and ``TestFlows``, you might want to limit
+your monitor events to only be applied on main tests and not sub-components
+(``run_monitor`` already behaves that way by default). For your convenience, you
+can use the following decorators on the overridden event methods to limit their activity:
+
+.. autofunction:: rotest.core.result.monitor.monitor.skip_if_case
+
+.. autofunction:: rotest.core.result.monitor.monitor.skip_if_flow
+
+.. autofunction:: rotest.core.result.monitor.monitor.skip_if_block
+
+.. autofunction:: rotest.core.result.monitor.monitor.skip_if_not_main
+
+.. autofunction:: rotest.core.result.monitor.monitor.require_attr

docs/cli_options/client_options.rst

@@ -428,3 +428,45 @@ with the output handlers separated using commas:
     $ rotest some_test_file.py --outputs excel,logdebug
 
 For more about output handlers, read on :ref:`output_handlers`.
+
+Adding New Options
+==================
+
+You can create new CLI options and behavior using the two entrypoints:
+``cli_client_parsers`` and ``cli_client_actions``.
+
+For example:
+
+.. code-block:: python
+
+    # utils/baz.py
+
+    def add_baz_option(parser):
+        """Add the 'baz' flag to the CLI options."""
+        parser.add_argument("--baz", "-B", action="store_true",
+                            help="The amazing Baz flag")
+
+
+    def use_baz_option(tests, config):
+        """Print the list of tests if 'baz' is on."""
+        if config.baz is True:
+            print tests
+
+
+And in your ``setup.py`` file inside ``setup()``:
+
+    .. code-block:: python
+
+        entry_points={
+            "rotest.cli_client_parsers":
+                ["baz_parser = utils.baz:add_baz_option"],
+            "rotest.cli_client_actions":
+                ["baz_func = utils.baz:use_baz_option"]
+        },
+
+
+* Make sure it's being installed in the environment by calling
+
+    .. code-block:: console
+
+        python setup.py develop

docs/index.rst

@@ -20,8 +20,9 @@ Advanced
 .. toctree::
     :maxdepth: 2
 
-    advanced/custom_output_handlers
     advanced/blocks
+    advanced/custom_output_handlers
+    advanced/monitors
 
 Indices and tables
 ==================

setup.py

@@ -2,7 +2,7 @@
 from __future__ import absolute_import
 from setuptools import setup, find_packages
 
-__version__ = "5.1.1"
+__version__ = "5.1.2"
 
 result_handlers = [
     "db = rotest.core.result.handlers.db_handler:DBHandler",

src/rotest/cli/client.py

@@ -214,7 +214,7 @@ def main(*tests):
         config.paths = (main_module,)
 
     if len(tests) == 0:
-        tests = discover_tests_under_paths(config.paths)
+        tests = list(discover_tests_under_paths(config.paths))
 
     if config.filter is not None:
         tests = [test for test in tests

src/rotest/core/__init__.py

@@ -1,7 +1,10 @@
-from .flow import TestFlow
-from .case import TestCase
-from .block import TestBlock
-from .suite import TestSuite
-from .abstract_test import request
-from .flow_component import (MODE_CRITICAL, MODE_FINALLY, MODE_OPTIONAL,
-                             PipeTo, BlockInput, BlockOutput)
+from .flow import TestFlow
+from .case import TestCase
+from .block import TestBlock
+from .suite import TestSuite
+from .abstract_test import request
+from .flow_component import (MODE_CRITICAL, MODE_FINALLY, MODE_OPTIONAL,
+                             PipeTo, BlockInput, BlockOutput)
+from .result.monitor.monitor import (AbstractMonitor, AbstractResourceMonitor,
+                                     require_attr, skip_if_case, skip_if_flow,
+                                     skip_if_block, skip_if_not_main)

src/rotest/core/result/monitor/monitor.py

@@ -2,6 +2,15 @@
 
 Implement monitors and use them as output handlers to monitor background
 processes, statuses and resources.
+
+A monitor is an output handler that uses tests' attributes, like resources.
+There are two kinds of monitors:
+1. basic monitors - react to the test's event - start, finish, error, etc.
+2. cyclic monitors - run periodically in the background during the test.
+
+To implement a cyclic monitor, override 'run_monitor' and specify 'CYCLE',
+if not implemented, the monitor will be a basic one (cyclic monitors can
+react to tests' event too).
 """
 # pylint: disable=broad-except
 from __future__ import absolute_import
@@ -74,17 +83,20 @@ def wrapped_func(self, test, *args, **kwargs):
     return wrapped_func
 
 
-class AbstractMonitor(AbstractResultHandler):
-    """Abstract monitor class.
+def skip_if_not_main(func):
+    """Avoid running the method if the test is a TestBlock or sub-flow."""
+    @wraps(func)
+    def wrapped_func(self, test, *args, **kwargs):
+        if isinstance(test, TestCase) or \
+                (isinstance(test, TestFlow) and test.is_main):
+
+            return func(self, test, *args, **kwargs)
 
-    A monitor is an output handler that uses tests' attributes, like resources.
-    There are two kinds of monitors:
-    1. basic monitors - react to the test's event - start, finish, error, etc.
-    2. cyclic monitors - run periodically in the background during the test.
+    return wrapped_func
 
-    To implement a cyclic monitor, override 'run_monitor' and specify 'CYCLE',
-    if not implemented, the monitor will be a basic one (cyclic monitors can
-    react to tests' event too).
+
+class AbstractMonitor(AbstractResultHandler):
+    """Abstract monitor class.
 
     Attributes:
         CYCLE (number): sleep time in seconds between monitor runs.
@@ -121,6 +133,7 @@ def safe_run_monitor(self, test):
             test.logger.exception("Got an error while running monitor %r",
                                   self.NAME)
 
+    @skip_if_not_main
     def setup_finished(self, test):
         """Handle test start event - register the monitor.
 
@@ -133,23 +146,19 @@ def setup_finished(self, test):
 
             return
 
-        if isinstance(test, TestCase) or \
-                (isinstance(test, TestFlow) and test.is_main):
-            test.logger.debug("Registering monitor %r", self.NAME)
-            self._failed = False
-            MonitorServer.register_monitor(self, test)
+        test.logger.debug("Registering monitor %r", self.NAME)
+        self._failed = False
+        MonitorServer.register_monitor(self, test)
 
+    @skip_if_not_main
     def start_teardown(self, test):
         """Handle test teardown event - unregister the monitor.
 
         Args:
             test (object): test item instance.
         """
-        if isinstance(test, TestCase) or \
-                (isinstance(test, TestFlow) and test.is_main):
-
-            test.logger.debug("Unregistering monitor %r", self.NAME)
-            MonitorServer.unregister_monitor(self)
+        test.logger.debug("Unregistering monitor %r", self.NAME)
+        MonitorServer.unregister_monitor(self)
 
     def fail_test(self, test, message):
         """Add a monitor failure to the test without stopping it.
@@ -170,7 +179,7 @@ class AbstractResourceMonitor(AbstractMonitor):
     """Abstract cyclic monitor that depends on a resource to run.
 
     This class extends the AbstractMonitor behavior and also waits for the
-    resource to be ready for work before starting the monitoring process.
+    resource to be ready for work before calling `run_monitor`.
 
     Attributes:
         RESOURCE_NAME (str): expected field name of the resource in the test.