Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/regression_test_api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,88 @@ In essence, these builtins exert control over the test creation, and they allow
:class:`reframe.core.fields.Field`.


Directives
----------

.. versionadded:: 3.5.3

Directives are special functions that are called at the class level but will be applied to the newly created test.
Directives can also be invoked as normal test methods once the test has been created.
Using directives and `builtins <#builtins>`__ together, it is possible to completely get rid of the :func:`__init__` method in the tests.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already true with the post-init hook, right? I think we should just avoid mentioning
__init__ in the new docs and just refer to the recommended syntax.

Static test information can be defined in the test class body and any adaptations based on the current system and/or environment can be made inside `pipeline hooks <#pipeline-hooks>`__.

.. py:function:: RegressionTest.depends_on(target, how=None, *args, **kwargs)

.. versionadded:: 2.21 (as a test method)

Add a dependency to another test.

:arg target: The name of the test that this one will depend on.
:arg how: A callable that defines how the test cases of this test depend on the the test cases of the target test. This callable should accept two arguments:

- The source test case (i.e., a test case of this test) represented as a two-element tuple containing the names of the partition and the environment of the current test case.
- Test destination test case (i.e., a test case of the target test) represented as a two-element tuple containing the names of the partition and the environment of the current target test case.

It should return :class:`True` if a dependency between the source and destination test cases exists, :class:`False` otherwise.

This function will be called multiple times by the framework when the test DAG is constructed, in order to determine the connectivity of the two tests.

In the following example, test ``T1`` depends on ``T0`` when their partitions match, otherwise their test cases are independent.

.. code-block:: python

def by_part(src, dst):
p0, _ = src
p1, _ = dst
return p0 == p1

class T1(rfm.RegressionTest):
depends_on('T0', how=by_part)

The framework offers already a set of predefined relations between the test cases of inter-dependent tests. See the :mod:`reframe.utility.udeps` for more details.

The default ``how`` function is :func:`~reframe.utility.udeps.by_case`, where test cases on different partitions and environments are independent.

.. seealso::
- :doc:`dependencies`
- :ref:`test-case-deps-management`

.. versionchanged:: 3.3
Dependencies between test cases from different partitions are now allowed. The ``how`` argument now accepts a callable.

.. deprecated:: 3.3
Passing an integer to the ``how`` argument as well as using the ``subdeps`` argument is deprecated.

.. versionchanged:: 3.5.3
This function has become a directive.


.. py:function:: RegressionTest.skip(msg=None)

Skip test.

:arg msg: A message explaining why the test was skipped.

.. versionadded:: 3.5.1

.. versionchanged:: 3.5.3
This function has become a directive.


.. py:function:: RegressionTest.skip_if(cond, msg=None)

Skip test if condition is true.

:arg cond: The condition to check for skipping the test.
:arg msg: A message explaining why the test was skipped.

.. versionadded:: 3.5.1

.. versionchanged:: 3.5.3
This function has become a directive.



Environments and Systems
------------------------

Expand Down
1 change: 1 addition & 0 deletions reframe/core/deferrable.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
def deferrable(func):
'''Function decorator for converting a function to a deferred
expression.'''

@functools.wraps(func)
def _deferred(*args, **kwargs):
return _DeferredExpression(func, *args, **kwargs)
Expand Down
76 changes: 76 additions & 0 deletions reframe/core/directives.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Copyright 2016-2021 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
# ReFrame Project Developers. See the top-level LICENSE file for details.
#
# SPDX-License-Identifier: BSD-3-Clause

#
# Base classes to manage the namespace of a regression test.
#

'''ReFrame Directives

A directive makes available a method defined in a class to the execution of
the class body during its creation.

A directive simply captures the arguments passed to it and all directives are
stored in a registry inside the class. When the final object is created, they
will be applied to that instance by calling the target method on the freshly
created object.
'''

NAMES = ('depends_on', 'skip', 'skip_if')


class _Directive:
'''A test directive.

A directive captures the arguments passed to it, so as to call an actual
object function later on during the test's initialization.

'''

def __init__(self, name, *args, **kwargs):
self._name = name
self._args = args
self._kwargs = kwargs

def __repr__(self):
cls = type(self).__qualname__
return f'{cls}({self.name!r}, {self.args}, {self.kwargs})'

@property
def name(self):
return self._name

@property
def args(self):
return self._args

@property
def kwargs(self):
return self._kwargs

def apply(self, obj):
fn = getattr(obj, self.name)
fn(*self.args, **self.kwargs)


class DirectiveRegistry:
def __init__(self):
self.__directives = []

@property
def directives(self):
return self.__directives

def add(self, name, *args, **kwargs):
self.__directives.append(_Directive(name, *args, **kwargs))


def apply(cls, obj):
'''Apply all directives of class ``cls`` to the object ``obj``.'''

for c in cls.mro():
if hasattr(c, '_rfm_dir_registry'):
for d in c._rfm_dir_registry.directives:
d.apply(obj)
16 changes: 15 additions & 1 deletion reframe/core/meta.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,13 @@
# Meta-class for creating regression tests.
#

import functools

import reframe.core.directives as directives
import reframe.core.namespaces as namespaces
import reframe.core.parameters as parameters
import reframe.core.variables as variables
from reframe.core.exceptions import ReframeSyntaxError

from reframe.core.exceptions import ReframeSyntaxError
from reframe.core.hooks import HookRegistry
Expand Down Expand Up @@ -116,6 +119,14 @@ def __prepare__(metacls, name, bases, **kwargs):
# Directives to add/modify a regression test variable
namespace['variable'] = variables.TestVar
namespace['required'] = variables.Undefined

# Insert the directives
dir_registry = directives.DirectiveRegistry()
for fn_name in directives.NAMES:
fn_dir_add = functools.partial(dir_registry.add, fn_name)
namespace[fn_name] = fn_dir_add

namespace['_rfm_dir_registry'] = dir_registry
return metacls.MetaNamespace(namespace)

def __new__(metacls, name, bases, namespace, **kwargs):
Expand Down Expand Up @@ -150,7 +161,7 @@ def __init__(cls, name, bases, namespace, **kwargs):
if hasattr(b, '_rfm_pipeline_hooks'):
hooks.update(getattr(b, '_rfm_pipeline_hooks'))

cls._rfm_pipeline_hooks = hooks # HookRegistry(local_hooks)
cls._rfm_pipeline_hooks = hooks
cls._final_methods = {v.__name__ for v in namespace.values()
if hasattr(v, '_rfm_final')}

Expand All @@ -162,6 +173,9 @@ def __init__(cls, name, bases, namespace, **kwargs):
return

for v in namespace.values():
if not hasattr(v, '__name__'):
continue

for b in bases:
if not hasattr(b, '_final_methods'):
continue
Expand Down
122 changes: 32 additions & 90 deletions reframe/core/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import os
import shutil

import reframe.core.directives as directives
import reframe.core.environments as env
import reframe.core.fields as fields
import reframe.core.hooks as hooks
Expand Down Expand Up @@ -772,6 +773,9 @@ def __new__(cls, *args, _rfm_use_params=False, **kwargs):

# Initialize the test
obj.__rfm_init__(name, prefix)

# Apply the directives
directives.apply(cls, obj)
return obj

def __init__(self):
Expand All @@ -789,11 +793,13 @@ def _add_hooks(cls, stage):
pipeline_hooks = cls._rfm_pipeline_hooks
fn = getattr(cls, stage)
new_fn = hooks.attach_hooks(pipeline_hooks)(fn)
setattr(cls, '_rfm_pipeline_fn_' + stage, new_fn)
setattr(cls, '_P_' + stage, new_fn)

def __getattribute__(self, name):
if name in _PIPELINE_STAGES:
name = f'_rfm_pipeline_fn_{name}'
name = f'_P_{name}'
elif name in directives.NAMES:
name = f'_D_{name}'

return super().__getattribute__(name)

Expand Down Expand Up @@ -1709,82 +1715,6 @@ def exact(src, dst):
else:
raise ValueError(f"unknown value passed to 'how' argument: {how}")

def depends_on(self, target, how=None, *args, **kwargs):
'''Add a dependency to another test.

:arg target: The name of the test that this one will depend on.
:arg how: A callable that defines how the test cases of this test
depend on the the test cases of the target test.
This callable should accept two arguments:

- The source test case (i.e., a test case of this test)
represented as a two-element tuple containing the names of the
partition and the environment of the current test case.
- Test destination test case (i.e., a test case of the target
test) represented as a two-element tuple containing the names of
the partition and the environment of the current target test
case.

It should return :class:`True` if a dependency between the source
and destination test cases exists, :class:`False` otherwise.

This function will be called multiple times by the framework when
the test DAG is constructed, in order to determine the
connectivity of the two tests.

In the following example, this test depends on ``T1`` when their
partitions match, otherwise their test cases are independent.

.. code-block:: python

def by_part(src, dst):
p0, _ = src
p1, _ = dst
return p0 == p1

self.depends_on('T0', how=by_part)

The framework offers already a set of predefined relations between
the test cases of inter-dependent tests. See the
:mod:`reframe.utility.udeps` for more details.

The default ``how`` function is
:func:`reframe.utility.udeps.by_case`, where test cases on
different partitions and environments are independent.

.. seealso::
- :doc:`dependencies`
- :ref:`test-case-deps-management`



.. versionadded:: 2.21

.. versionchanged:: 3.3
Dependencies between test cases from different partitions are now
allowed. The ``how`` argument now accepts a callable.

.. deprecated:: 3.3
Passing an integer to the ``how`` argument as well as using the
``subdeps`` argument is deprecated.

'''
if not isinstance(target, str):
raise TypeError("target argument must be of type: `str'")

if (isinstance(how, int)):
# We are probably using the old syntax; try to get a
# proper how function
how = self._depends_on_func(how, *args, **kwargs)

if how is None:
how = udeps.by_case

if not callable(how):
raise TypeError("'how' argument must be callable")

self._userdeps.append((target, how))

def getdep(self, target, environ=None, part=None):
'''Retrieve the test case of a target dependency.

Expand Down Expand Up @@ -1823,23 +1753,35 @@ def getdep(self, target, environ=None, part=None):
raise DependencyError(f'could not resolve dependency to ({target!r}, '
f'{part!r}, {environ!r})')

def skip(self, msg=None):
'''Skip test.
# Directives

:arg msg: A message explaining why the test was skipped.
def _D_depends_on(self, target, how=None, *args, **kwargs):
'''Add a dependency to another test.'''

.. versionadded:: 3.5.1
'''
raise SkipTestError(msg)
if not isinstance(target, str):
raise TypeError("target argument must be of type: `str'")

if (isinstance(how, int)):
# We are probably using the old syntax; try to get a
# proper how function
how = self._depends_on_func(how, *args, **kwargs)

if how is None:
how = udeps.by_case

if not callable(how):
raise TypeError("'how' argument must be callable")

def skip_if(self, cond, msg=None):
'''Skip test if condition is true.
self._userdeps.append((target, how))

:arg cond: The condition to check for skipping the test.
:arg msg: A message explaining why the test was skipped.
def _D_skip(self, msg=None):
'''Skip test.'''

raise SkipTestError(msg)

def _D_skip_if(self, cond, msg=None):
'''Skip test if condition is true.'''

.. versionadded:: 3.5.1
'''
if cond:
self.skip(msg)

Expand Down
Loading