[feat] Add sanity_function and deferrable builtins

cscs-checks/system/jobreport/gpu_report.py

@@ -43,9 +43,9 @@ def set_launcher_opts(self):
         '''
         self.job.launcher.options = ['-u']
 
-    @run_before('sanity')
+    @sanity_function
     def set_sanity_patterns(self):
-        '''Set sanity patterns and wait for the jobreport.
+        '''Extend sanity and wait for the jobreport.
 
         If a large number of nodes is used, the final jobreport output happens
         much later after job has already completed (this could be up to 25s).
@@ -54,16 +54,16 @@ def set_sanity_patterns(self):
         sanity function does not succeed.
         '''
 
-        super().set_sanity_patterns()
-        self.sanity_patterns = sn.all([
-            self.sanity_patterns, self.gpu_usage_sanity()
-        ])
         try:
             sn.evaluate(self.gpu_usage_sanity())
         except SanityError:
             time.sleep(25)
 
-    @sn.sanity_function
+        return sn.all([
+            self.count_successful_burns(), self.gpu_usage_sanity()
+        ])
+
+    @deferrable
     def gpu_usage_sanity(self):
         '''Verify that the jobreport output has sensible numbers.
 

docs/deferrable_functions_reference.rst

@@ -0,0 +1,79 @@
+ .. _deferrable-functions:
+
+==============================
+Deferrable Functions Reference
+==============================
+
+*Deferrable functions* are the functions whose execution may be postponed to a later time after they are called.
+The key characteristic of these functions is that they store their arguments when they are called, and the execution itself does not occur until the function is evaluated either explicitly or implicitly.
+
+Explicit evaluation of deferrable functions
+-------------------------------------------
+
+Deferrable functions may be evaluated at any time by calling :func:`evaluate` on their return value or by passing the deferred function itself to the :func:`~reframe.utility.sanity.evaluate()` free function.
+
+Implicit evaluation of deferrable functions
+-------------------------------------------
+
+Deferrable functions may also be evaluated implicitly in the following situations:
+
+- When you try to get their truthy value by either explicitly or implicitly calling :func:`bool <python:bool>` on their return value.
+  This implies that when you include the result of a deferrable function in an :keyword:`if` statement or when you apply the :keyword:`and`, :keyword:`or` or :keyword:`not` operators, this will trigger their immediate evaluation.
+
+- When you try to iterate over their result.
+  This implies that including the result of a deferrable function in a :keyword:`for` statement will trigger its evaluation immediately.
+
+- When you try to explicitly or implicitly get its string representation by calling :func:`str <python:str>` on its result.
+  This implies that printing the return value of a deferrable function will automatically trigger its evaluation.
+
+
+Categories of deferrable functions
+----------------------------------
+
+Currently ReFrame provides three broad categories of deferrable functions:
+
+1. Deferrable replacements of certain Python built-in functions.
+   These functions simply delegate their execution to the actual built-ins.
+2. Assertion functions.
+   These functions are used to assert certain conditions and they either return :class:`True` or raise :class:`~reframe.core.exceptions.SanityError` with a message describing the error.
+   Users may provide their own formatted messages through the ``msg`` argument.
+   For example, in the following call to :func:`assert_eq` the ``{0}`` and ``{1}`` placeholders will obtain the actual arguments passed to the assertion function.
+
+   .. code:: python
+
+        assert_eq(a, 1, msg="{0} is not equal to {1}")
+
+   If in the user provided message more placeholders are used than the arguments of the assert function (except the ``msg`` argument), no argument substitution will be performed in the user message.
+3. Utility functions.
+   They include, but are not limited to, functions to iterate over regex matches in a file, extracting and converting values from regex matches, computing statistical information on series of data etc.
+
+
+Users can write their own deferrable functions as well.
+The page ":doc:`deferrables`" explains in detail how deferrable functions work and how users can write their own.
+
+
+.. py:decorator:: reframe.utility.sanity.deferrable(func)
+
+    Deferrable decorator.
+
+    Converts the decorated free function into a deferrable function.
+
+    .. code:: python
+
+        import reframe.utility.sanity as sn
+
+        @sn.deferrable
+        def myfunc(*args):
+            do_sth()
+
+
+.. py:decorator:: reframe.utility.sanity.sanity_function(func)
+
+    Please use the :func:`reframe.core.pipeline.RegressionMixin.deferrable` decorator when possible. Alternatively, please use the :func:`reframe.utility.sanity.deferrable` decorator instead.
+
+    .. warning:: Not to be mistaken with :func:`~reframe.core.pipeline.RegressionMixin.sanity_function` built-in.
+    .. deprecated:: 3.8.0
+
+
+.. automodule:: reframe.utility.sanity
+    :members:

docs/deferrables.rst

@@ -1,23 +1,24 @@
-===============================================
-Understanding the Mechanism of Sanity Functions
-===============================================
+===================================================
+Understanding the Mechanism of Deferrable Functions
+===================================================
 
-This section describes the mechanism behind the sanity functions that are used for the sanity and performance checking.
-Generally, writing a new sanity function is as straightforward as decorating a simple Python function with the :func:`reframe.utility.sanity.sanity_function` decorator.
-However, it is important to understand how and when a deferrable function is evaluated, especially if your function takes as arguments the results of other deferrable functions.
+This section describes the mechanism behind deferrable functions, which in ReFrame, they are used for sanity and performance checking.
+Generally, writing a new sanity function in a :class:`~reframe.core.pipeline.RegressionTest` is as straightforward as decorating a simple member function with the built-in :func:`~reframe.core.pipeline.RegressionMixin.sanity_function` decorator.
+Behind the scenes, this decorator will convert the Python function into a deferrable function and schedule its evaluation for the sanity stage of the test.
+However, when dealing with more complex scenarios such as a deferrable function taking as an argument the results from other deferrable functions, it is crucial to understand how a deferrable function differs from a regular Python function, and when is it actually evaluated.
 
 What Is a Deferrable Function?
 ------------------------------
 
 A deferrable function is a function whose a evaluation is deferred to a later point in time.
-You can define any function as deferrable by wrapping it with the :func:`reframe.utility.sanity.sanity_function` decorator before its definition.
+You can define any function as deferrable by wrapping it with the :func:`~reframe.core.pipeline.RegressionMixin.deferrable` when decorating a member function of a class derived from :class:`~reframe.core.pipeline.RegressionMixin`, or alternatively, the :func:`reframe.utility.sanity.deferrable` decorator can be used for any other function.
 The example below demonstrates a simple scenario:
 
 .. code-block:: python
 
   import reframe.utility.sanity as sn
 
-  @sn.sanity_function
+  @sn.deferrable
   def foo():
       print('hello')
 
@@ -47,11 +48,11 @@ Deferrable functions may also be combined as we do with normal functions. Let's
 
   import reframe.utility.sanity as sn
 
-  @sn.sanity_function
+  @sn.deferrable
   def foo(arg):
       print(arg)
 
-  @sn.sanity_function
+  @sn.deferrable
   def greetings():
       return 'hello'
 
@@ -85,7 +86,7 @@ To demonstrate more clearly how the deferred evaluation of a function works, let
 
 .. code-block:: python
 
-  @sn.sanity_function
+  @sn.deferrable
   def size3(iterable):
       return len(iterable) == 3
 
@@ -160,7 +161,7 @@ Here is why:
 
 .. code-block:: pycon
 
-  >>> @sn.sanity_function
+  >>> @sn.deferrable
   ... def size(iterable):
   ...     return len(iterable)
   ...
@@ -184,7 +185,7 @@ If you want to defer the execution of such operators, you should use the corresp
 
 In summary deferrable functions have the following characteristics:
 
-* You can make any function deferrable by wrapping it with the :func:`reframe.utility.sanity.sanity_function` decorator.
+* You can make any function deferrable by wrapping it with the :func:`~reframe.utility.sanity.deferrable` decorator.
 * When you call a deferrable function, its body is not executed but its arguments are *captured* and an object representing the deferred function is returned.
 * You can execute the body of a deferrable function at any later point by calling :func:`evaluate <reframe.utility.sanity.evaluate>` on the deferred expression object that it has been returned by the call to the deferred function.
 * Deferred functions can accept other deferred expressions as arguments and may also return a deferred expression.
@@ -194,7 +195,7 @@ In summary deferrable functions have the following characteristics:
 How a Deferred Expression Is Evaluated?
 ---------------------------------------
 
-As discussed before, you can create a new deferred expression by calling a function whose definition is decorated by the ``@sanity_function`` or ``@deferrable`` decorator or by including an already deferred expression in any sort of arithmetic operation.
+As discussed before, you can create a new deferred expression by calling a function whose definition is decorated by the ``@deferrable`` decorator or by including an already deferred expression in any sort of arithmetic operation.
 When you call :func:`evaluate <reframe.utility.sanity.evaluate>` on a deferred expression, you trigger the evaluation of the whole subexpression tree.
 Here is how the evaluation process evolves:
 
@@ -209,15 +210,15 @@ Here is an example where we define two deferrable variations of the builtins :fu
 
 .. code-block:: python
 
-  @sn.sanity_function
+  @sn.deferrable
   def dsum(iterable):
       return sum(iterable)
 
-  @sn.sanity_function
+  @sn.deferrable
   def dlen(iterable):
       return len(iterable)
 
-  @sn.sanity_function
+  @sn.deferrable
   def avg(iterable):
       return dsum(iterable) / dlen(iterable)
 
@@ -285,7 +286,7 @@ Although you can trigger the evaluation of a deferred expression at any time by
 
   .. code-block:: pycon
 
-    >>> @sn.sanity_function
+    >>> @sn.deferrable
     ... def getlist(iterable):
     ...     ret = list(iterable)
     ...     ret += [1, 2, 3]
@@ -336,12 +337,12 @@ The following example demonstrates two different ways writing a deferrable funct
 
   import reframe.utility.sanity as sn
 
-  @sn.sanity_function
+  @sn.deferrable
   def check_avg_with_deferrables(iterable):
       avg = sn.sum(iterable) / sn.len(iterable)
       return -1 if avg > 2 else 1
 
-  @sn.sanity_function
+  @sn.deferrable
   def check_avg_without_deferrables(iterable):
       avg = sum(iterable) / len(iterable)
       return -1 if avg > 2 else 1
@@ -360,14 +361,14 @@ In the version with the deferrables, ``avg`` is a deferred expression but it is
 
 Generally, inside a sanity function, it is a preferable to use the non-deferrable version of a function, if that exists, since you avoid the extra overhead and bookkeeping of the deferring mechanism.
 
-Deferrable Sanity Functions
----------------------------
+Ready to Go Deferrable Functions
+--------------------------------
 
-Normally, you will not have to implement your own sanity functions, since ReFrame provides already a variety of them.
-You can find the complete list of provided sanity functions `here <sanity_functions_reference.html>`__.
+Normally, you will not have to implement your own deferrable functions, since ReFrame provides already a variety of them.
+You can find the complete list of provided sanity functions in :ref:`deferrable-functions`.
 
-Similarities and Differences with Generators
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Deferrable functions vs Generators
+----------------------------------
 
 Python allows you to create functions that will be evaluated lazily.
 These are called `generator functions <https://wiki.python.org/moin/Generators>`__.
@@ -390,7 +391,7 @@ Differences
 
   .. code-block:: pycon
 
-    >>> @sn.sanity_function
+    >>> @sn.deferrable
     ... def dsize(iterable):
     ...     print(len(iterable))
     ...     return len(iterable)

docs/programming_apis.rst

@@ -6,6 +6,6 @@ Programming APIs
    :maxdepth: 3
 
    regression_test_api
-   sanity_functions_reference
+   deferrable_functions_reference
    utility_functions_reference
    exceptions

docs/regression_test_api.rst

@@ -262,13 +262,36 @@ The framework will then continue with other activities and it will execute the p
 Built-in functions
 ------------------
 
+.. py:decorator:: RegressionMixin.sanity_function(func)
+
+  Decorate a member function as the sanity function of the test.
+
+  This decorator will convert the decorated method into a :func:`~RegressionMixin.deferrable` and mark it to be executed during the test's sanity stage.
+  When this decorator is used, manually assigning a value to :attr:`~RegressionTest.sanity_patterns` in the test is not allowed.
+
+  Decorated functions may be overridden by derived classes, and derived classes may also decorate a different method as the test's sanity function.
+  Decorating multiple member functions in the same class is not allowed.
+  However, a :class:`RegressionTest` may inherit from multiple :class:`RegressionMixin` classes with their own sanity functions.
+  In this case, the derived class will follow Python's `MRO <https://docs.python.org/3/library/stdtypes.html#class.__mro__>`_ to find a suitable sanity function.
+
+  .. versionadded:: 3.7.0
+
+.. py:decorator:: RegressionMixin.deferrable(func)
+
+  Converts the decorated method into a deferrable function.
+
+  See :ref:`deferrable-functions` for further information on deferrable functions.
+
+  .. versionadded:: 3.7.0
+
 .. py:function:: RegressionMixin.bind(func, name=None)
 
   Bind a free function to a regression test.
+
   By default, the function is bound with the same name as the free function.
   However, the function can be bound using a different name with the ``name`` argument.
 
-  :param fn: external function to be bound to a class.
+  :param func: external function to be bound to a class.
   :param name: bind the function under a different name.
 
   .. versionadded:: 3.6.2

docs/tutorial_basics.rst

@@ -575,7 +575,7 @@ They will not be executed where they appear, but rather at the sanity checking p
 ReFrame provides lazily evaluated counterparts for most of the builtin Python functions, such the :func:`len` function here.
 Also whole expressions can be lazily evaluated if one of the operands is deferred, as is the case in this example with the assignment to ``num_messages``.
 This makes the sanity checking mechanism quite powerful and straightforward to reason about, without having to rely on complex pattern matching techniques.
-:doc:`sanity_functions_reference` provides a complete reference of the sanity functions provided by ReFrame, but users can also define their own, as described in :doc:`deferrables`.
+:doc:`deferrable_functions_reference` provides a complete reference of the sanity functions provided by ReFrame, but users can also define their own, as described in :doc:`deferrables`.
 
 
 Let's run this version of the test now and see if it fails: