Skip to content

[doc] Update tutorials to the new sanity and performance syntax #2167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Sep 16, 2021
Merged

[doc] Update tutorials to the new sanity and performance syntax #2167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
83254e9
Update basic tutorials
jjotero Aug 30, 2021
bd2e562
Update advanced tutorials
jjotero Aug 30, 2021
d22ad54
Update dependencies tutorials
jjotero Aug 30, 2021
3193c52
Update tips and tricks
jjotero Aug 30, 2021
dc17489
Remove other references to perf_patterns and sanity_patterns
jjotero Aug 30, 2021
15f833d
Bugfix hello OMP
jjotero Aug 30, 2021
da60575
Address PR comments
jjotero Sep 1, 2021
d7c8d14
Fix typo in the docs
jjotero Sep 1, 2021
2c794d1
Address comments
jjotero Sep 9, 2021
5793df1
Address PR comments
Sep 13, 2021
4eb22aa
Add stream4 file
jjotero Sep 14, 2021
648d621
Merge branch 'master' into docs/syntax
Sep 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configure.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ You can view logger's log level as a general cut off.
For example, if we have set it to ``warning``, no debug or informational messages would ever be printed.

Finally, there is a special set of handlers for handling performance log messages.
Performance log messages are generated *only* for `performance tests <tutorial_basics.html#writing-a-performance-test>`__, i.e., tests defining the :attr:`perf_patterns <reframe.core.pipeline.RegressionTest.perf_patterns>` attribute.
Performance log messages are generated *only* for `performance tests <tutorial_basics.html#writing-a-performance-test>`__, i.e., tests defining the :attr:`~reframe.core.pipeline.RegressionTest.perf_variables` or the :attr:`~reframe.core.pipeline.RegressionTest.perf_patterns` attributes.
The performance log handlers are stored in the ``handlers_perflog`` property.
The ``filelog`` handler used in this example will create a file per test and per system/partition combination (``./<system>/<partition>/<testname>.log``) and will append to it the obtained performance data every time a performance test is run.
Notice how the message to be logged is structured in the ``format`` property, such that it can be easily parsed from post processing tools.
Expand Down
40 changes: 18 additions & 22 deletions docs/tutorial_advanced.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,8 @@ Let's have a look at the test itself:


.. literalinclude:: ../tutorials/advanced/makefiles/maketest.py
:lines: 6-29
:emphasize-lines: 18,22-24
:lines: 6-27
:emphasize-lines: 18,22

First, if you're using any build system other than ``SingleSource``, you must set the :attr:`executable` attribute of the test, because ReFrame cannot know what is the actual executable to be run.
We then set the build system to :class:`~reframe.core.buildsystems.Make` and set the preprocessor flags as we would do with the :class:`SingleSource` build system.
Expand All @@ -161,7 +161,7 @@ Let's inspect the build script generated by ReFrame:

trap _onerror ERR

make -j 1 CPPFLAGS="-DELEM_TYPE=float"
make -j 1 CC="cc" CXX="CC" FC="ftn" NVCC="nvcc" CPPFLAGS="-DELEM_TYPE=float"


The compiler variables (``CC``, ``CXX`` etc.) are set based on the corresponding values specified in the `configuration <config_reference.html#environment-configuration>`__ of the current environment.
Expand Down Expand Up @@ -249,7 +249,7 @@ Writing a Run-Only Regression Test
----------------------------------

There are cases when it is desirable to perform regression testing for an already built executable.
In the following test we use simply the ``echo`` Bash shell command to print a random integer between specific lower and upper bounds:
In the following test we use simply the ``echo`` Bash shell command to print a random integer between specific lower and upper bounds.
Here is the full regression test:

.. code-block:: console
Expand Down Expand Up @@ -281,7 +281,7 @@ The following test is a compile-only version of the :class:`MakefileTest` presen


.. literalinclude:: ../tutorials/advanced/makefiles/maketest.py
:lines: 32-
:lines: 30-
:emphasize-lines: 2

What is worth noting here is that the standard output and standard error of the test, which are accessible through the :attr:`~reframe.core.pipeline.RegressionTest.stdout` and :attr:`~reframe.core.pipeline.RegressionTest.stderr` attributes, correspond now to the standard output and error of the compilation command.
Expand Down Expand Up @@ -337,7 +337,7 @@ Notice how the parameters are expanded in each of the individual tests:
Applying a Sanity Function Iteratively
--------------------------------------

It is often the case that a common sanity pattern has to be applied many times.
It is often the case that a common sanity function has to be applied many times.
The following script prints 100 random integers between the limits given by the environment variables ``LOWER`` and ``UPPER``.

.. code-block:: console
Expand Down Expand Up @@ -372,7 +372,7 @@ There is still a small complication that needs to be addressed.
As a direct replacement of the built-in :py:func:`all` function, ReFrame's :func:`~reframe.utility.sanity.all` sanity function returns :class:`True` for empty iterables, which is not what we want.
So we must make sure that all 100 numbers are generated.
This is achieved by the ``sn.assert_eq(sn.count(numbers), 100)`` statement, which uses the :func:`~reframe.utility.sanity.count` sanity function for counting the generated numbers.
Finally, we need to combine these two conditions to a single deferred expression that will be assigned to the test's :attr:`sanity_patterns`.
Finally, we need to combine these two conditions to a single deferred expression that will be returned by the test's :attr:`@sanity_function<reframe.core.pipeline.RegressionMixin.sanity_function>`.
We accomplish this by using the :func:`~reframe.utility.sanity.all` sanity function.

For more information about how exactly sanity functions work and how their execution is deferred, please refer to :doc:`deferrables`.
Expand Down Expand Up @@ -458,7 +458,7 @@ Here is the test:


.. literalinclude:: ../tutorials/advanced/jobopts/eatmemory.py
:lines: 6-23
:lines: 6-25
:emphasize-lines: 12-14

Each ReFrame test has an associated `run job descriptor <regression_test_api.html#reframe.core.pipeline.RegressionTest.job>`__ which represents the scheduler job that will be used to run this test.
Expand Down Expand Up @@ -492,7 +492,7 @@ Let's run the test and inspect the generated job script:
The job options specified inside a ReFrame test are always the last to be emitted in the job script preamble and do not affect the options that are passed implicitly through other test attributes or configuration options.

There is a small problem with this test though.
What if we change the job scheduler in that partition or what if we want to port the test to a different system that does not use Slurm and and another option is needed to achieve the same result.
What if we change the job scheduler in that partition or what if we want to port the test to a different system that does not use Slurm and another option is needed to achieve the same result.
The obvious answer is to adapt the test, but is there a more portable way?
The answer is yes and this can be achieved through so-called *extra resources*.
ReFrame gives you the possibility to associate scheduler options to a "resource" managed by the partition scheduler.
Expand Down Expand Up @@ -625,7 +625,7 @@ It resembles a scaling test, except that all happens inside a single ReFrame tes

.. literalinclude:: ../tutorials/advanced/multilaunch/multilaunch.py
:lines: 6-
:emphasize-lines: 12-19
:emphasize-lines: 13-19

The additional parallel launch commands are inserted in either the :attr:`prerun_cmds` or :attr:`postrun_cmds` lists.
To retrieve the actual parallel launch command for the current partition that the test is running on, you can use the :func:`~reframe.core.launchers.Launcher.run_command` method of the launcher object.
Expand Down Expand Up @@ -680,13 +680,9 @@ The test will verify that all the nodes print the expected host name:
:lines: 6-
:emphasize-lines: 10-

The first thing to notice in this test is that :attr:`~reframe.core.pipeline.RegressionTest.num_tasks` is set to zero.
This is a requirement for flexible tests.
The sanity check of this test simply counts the host names printed and verifies that they are as many as expected.
Notice, however, that the sanity check does not use :attr:`num_tasks` directly, but rather access the attribute through the :func:`~reframe.utility.sanity.getattr` sanity function, which is a replacement for the :func:`getattr` builtin.
The reason for that is that at the time the sanity check expression is created, :attr:`num_tasks` is ``0`` and it will only be set to its actual value during the run phase.
Consequently, we need to defer the attribute retrieval, thus we use the :func:`~reframe.utility.sanity.getattr` sanity function instead of accessing it directly

The first thing to notice in this test is that :attr:`~reframe.core.pipeline.RegressionTest.num_tasks` is set to zero as default, which is a requirement for flexible tests.
However, with flexible tests, this value is updated right after the job completes to the actual number of tasks that were used.
Consequently, this allows the sanity function of the test to assert that the number host names printed matches :attr:`~reframe.core.pipeline.RegressionTest.num_tasks`.

.. |--flex-alloc-nodes| replace:: :attr:`--flex-alloc-nodes`
.. _--flex-alloc-nodes: manpage.html#cmdoption-flex-alloc-nodes
Expand Down Expand Up @@ -723,7 +719,7 @@ The following parameterized test, will create two tests, one for each of the sup

.. literalinclude:: ../tutorials/advanced/containers/container_test.py
:lines: 6-
:emphasize-lines: 16-22
:emphasize-lines: 11-19

A container-based test can be written as :class:`~reframe.core.pipeline.RunOnlyRegressionTest` that sets the :attr:`~reframe.core.pipeline.RegressionTest.container_platform` attribute.
This attribute accepts a string that corresponds to the name of the container platform that will be used to run the container for this test.
Expand Down Expand Up @@ -780,7 +776,7 @@ and ``/rfm_workdir`` corresponds to the stage directory on the host system.
Therefore, the ``release.txt`` file can now be used in the subsequent sanity checks:

.. literalinclude:: ../tutorials/advanced/containers/container_test.py
:lines: 15-17
:lines: 26-29


For a complete list of the available attributes of a specific container platform, please have a look at the :ref:`container-platforms` section of the :doc:`regression_test_api` guide.
Expand All @@ -793,8 +789,8 @@ Writing reusable tests
.. versionadded:: 3.5.0

So far, all the examples shown above were tight to a particular system or configuration, which makes reusing these tests in other systems not straightforward.
However, the introduction of the :py:func:`~reframe.core.pipeline.RegressionTest.parameter` and :py:func:`~reframe.core.pipeline.RegressionTest.variable` ReFrame built-ins solves this problem, eliminating the need to specify any of the test variables in the :func:`__init__` method and simplifying code reuse.
Hence, readers who are not familiar with these built-in functions are encouraged to read their basic use examples (see :py:func:`~reframe.core.pipeline.RegressionTest.parameter` and :py:func:`~reframe.core.pipeline.RegressionTest.variable`) before delving any deeper into this tutorial.
However, the introduction of the :py:func:`~reframe.core.pipeline.RegressionMixin.parameter` and :py:func:`~reframe.core.pipeline.RegressionMixin.variable` ReFrame built-ins solves this problem, eliminating the need to specify any of the test variables in the :func:`__init__` method and simplifying code reuse.
Hence, readers who are not familiar with these built-in functions are encouraged to read their basic use examples (see :py:func:`~reframe.core.pipeline.RegressionMixin.parameter` and :py:func:`~reframe.core.pipeline.RegressionMixin.variable`) before delving any deeper into this tutorial.

In essence, parameters and variables can be treated as simple class attributes, which allows us to leverage Python's class inheritance and write more modular tests.
For simplicity, we illustrate this concept with the above :class:`ContainerTest` example, where the goal here is to re-write this test as a library that users can simply import from and derive their tests without having to rewrite the bulk of the test.
Expand All @@ -821,7 +817,7 @@ In fact, writing the test in this way permits having hooks that depend on undefi
This is the case with the :func:`set_container_platform` hook, which depends on the undefined parameter ``platform``.
Hence, the derived test **must** define all the required parameters and variables; otherwise ReFrame will notice that the test is not well defined and will raise an error accordingly.

Before moving onwards to the derived test, note that the :class:`ContainerBase` class takes the additional argument ``pin_prefix=True``, which locks the prefix of all derived tests to this base test.
Before moving ahead with the derived test, note that the :class:`ContainerBase` class takes the additional argument ``pin_prefix=True``, which locks the prefix of all derived tests to this base test.
This will allow the retrieval of the sources located in the library by any derived test, regardless of what their containing directory is.

.. code-block:: console
Expand Down
Loading