doc: update and clean the doc

CHANGELOG.rst

@@ -15,19 +15,15 @@ Changed
 ~~~~~~~
 - The name of the runner shell script in the output directories is the one
   passed to the CLI instead of the hardcoded one.
-- All the names of the CLI options have been prefixed with *exe* to prevent
-  from clashing with other plugins CLI names.
-- The settings that can be replaced in the runner script are now defined in a
-  dedicated section in test_case.yaml.
+- All the names of the CLI options have been prefixed with :option:`--exe-` to
+  prevent name clashes with other plugins options.
+- It is easier to define the settings to execute the runner shell script for a
+  test case thanks to a dedicated section in test_case.yaml.
 
 Added
 ~~~~~
 - Testing on MacOS.
 
-Fixed
-~~~~~
-- One test on MacOS.
-
 0.4.0 - 2020-05-03
 ------------------
 

README.rst

@@ -78,10 +78,8 @@ Apache License 2.0. See the `LICENSE.txt`_ file for more information, or the
 `quick summary`_ of this license on `tl;drLegal`_ website.
 
 
-.. _pytest: https://docs.pytest.org/en/latest
+.. _pytest: https://docs.pytest.org
 .. _online: https://pytest-executable.readthedocs.io
-.. _`doc`: doc
-.. _documentation: https://pytest-executable.readthedocs.io/en/stable/installation.html
 .. _Github issue tracker: https://github.com/CS-SI/pytest-executable/issues
 .. _Antoine Dechaume: https://github.com/AntoineD
 .. _CS GROUP: http://www.csgroup.eu

doc/conf.py

@@ -63,9 +63,10 @@
 autodoc_docstring_signature = False
 
 rst_prolog = """
+.. _pytest: https://docs.pytest.org
 .. |ptx| replace:: *pytest-executable*
 .. |exe| replace:: :program:`executable`
-.. |pytest| replace:: :program:`pytest`
+.. |pytest| replace:: `pytest`_
 .. |yaml| replace:: :file:`test_case.yaml`
 .. |runner| replace:: *runner shell script*
 """

doc/fixtures.rst

@@ -27,25 +27,16 @@ The purpose of test fixtures is to ease the writing of test functions by
 providing informations and data automatically. You may find more documentation
 on |pytest| fixture in its `official documentation
 <https://docs.pytest.org/en/latest/fixture.html>`_. We describe here the
-fixtures defined in |ptx|. They are used in the default test module, give a
-look at it for usage examples, see :ref:`builtin-test-module`.
-
+fixtures defined in |ptx|. Some of them are used in the default test module,
+see :ref:`builtin-test-module`.
 
 Runner fixture
 --------------
 
-This fixture is used to run |exe|, it will do the following:
-
-- get the runner script passed to the |pytest| command line option
-  :option:`--exe-runner`,
-- process it to replace the placeholders `{{nproc}}` and `{{output_path}}` with their
-  actual values,
-- write it to the |runner| in the test case output directory.
-
-The :py:data:`runner` object provided by the fixture can be executed with the
-:py:meth:`run` method which will return the exit status of the script
-execution. The value **0** of the exit status means a successful execution.
-
+This fixture is used to execute |runner|. The :py:data:`runner` object provided
+by the fixture can be executed with the :py:meth:`run` method which will return
+the exit status of the script execution. The value **0** of the exit status
+means a successful execution.
 
 Output path fixture
 -------------------
@@ -54,25 +45,34 @@ This fixture is used to get the absolute path to the output directory of a test
 case. It provides the :py:data:`output_path` variable that holds a `Path`_
 object.
 
-
-Tolerances fixture
-------------------
-
-This fixture is used to get the values of the tolerances defined in the |yaml|.
-It provides the :py:data:`tolerances` dictionary that binds the name of a
-quantity to an object that has 2 attributes:
-
-- :py:attr:`rel`: the relative tolerance,
-- :py:attr:`abs`: the absolute tolerance.
-
+.. _regression-path-fixtures:
 
 Regression path fixture
 -----------------------
 
-This fixture is used to get the absolute path to the directory that contains
-the regression reference of a test case when the command line option
-:option:`--exe-regression-root` is used. It provides the :py:data:`regression_path`
-variable that holds a `Path`_ object.
+This fixture is used to get the paths to the reference data of a test case, see
+:ref:`yaml-ref`. If :option:`--exe-regression-root` is not used then the test
+functions that use the fixture are skipped. Otherwise, the test functions that
+use this fixture are called once per reference item (file or directory)
+declared in the references section of |yaml|. The fixture name is
+:py:data:`regression_file_path`, it's an object with the attributes:
+
+- :py:attr:`relative`: a `Path`_ object that contains the path to a reference
+  item relatively to the test case output directory.
+- :py:attr:`absolute`: a `Path`_ object that contains the absolute path to a
+  reference item.
 
 You may use this fixture with the :py:data:`output_path` fixture to get the
 path to the file that shall be compared to a reference file.
+
+.. _tolerances-fixtures:
+
+Tolerances fixture
+------------------
+
+This fixture is used to get the values of the tolerances defined in the |yaml|,
+see :ref:`yaml-tol`. It provides the :py:data:`tolerances` dictionary that
+binds a data name to an object that has 2 attributes:
+
+- :py:attr:`rel`: the relative tolerance,
+- :py:attr:`abs`: the absolute tolerance.

doc/how-to-use.rst

@@ -58,7 +58,7 @@ Then the tests output tree is::
    │   ├── runner.sh
    │   ├── runner.sh.stderr
    │   └── runner.sh.stdout
-   ├── dir-2
+   └── dir-2
        ├── input -> path/to/tests/inputs/dir-2/input
        ├── output
        ├── executable.stderr

doc/test-case.rst

@@ -20,101 +20,142 @@
 Add a test case
 ===============
 
-A test case is composed of a directory with:
+A test case is composed of an input directory with:
 
-- the input files of the |exe|
-- a |yaml| file with |ptx| settings
-- optionnal |pytest| and python modules for additionnal tests
+- the input files required by the |runner|,
+- a |yaml| file with the |ptx| settings,
+- any optionnal |pytest| python modules for performing additionnal tests.
 
-Among the optionnal modules, there shall be at least one that is discoverable
-by |pytest|, i.e. a python module which name starts with :file:`test_` and
-which contains at least one function which also starts with **test_**.
+.. warning::
 
-.. note::
-
-   A test case directory shall not contain any of the files created by the
-   execution of the |exe| or of the processing defined in the python modules,
-   otherwise they may badly interfere with the execution of the testing tool.
-   In other words: do not run the |exe| in the input directory.
+   An test case input directory shall not contain any of the files created by
+   the execution of the |exe| or of the additional python modules, otherwise
+   they may badly interfere with the executions done by |ptx|. In other words:
+   do not run anything in the input directory of a test case, this directory
+   shall only contain input data.
 
 The |yaml| file is used by |ptx| for several things. When this file is
-found, |ptx| will create the test case output directory, then identify the
-settings for running the case and finally perform the checks and
-post-porcesses. If |yaml| is empty, then the default settings are used, which
-is equivalent to using a |yaml| with the following contents:
+found, |ptx| will:
+
+   1. create the test case output directory and, if needed, its parents,
+   2. read the settings for running the test case,
+   3. go to the test case output directory and execute the tests defined
+   4. in the default test module,
+   5. then in the additional test modules.
+
+The parents of the output directory are created such that the path from the
+directory where |pytest| is executed to the test case input directory is the
+same but for the first parent. This way, the directories hierarchy below the
+first parent of both the inputs and the outputs trees are the same.
+
+If |yaml| is empty, then the default settings are used. If
+:option:`--exe-default-settings` is not set, the default settings are the
+builtin ones:
 
  .. literalinclude:: ../src/pytest_executable/test_case.yaml
 
-This file is in yaml format, a widely used human friendly file format that
-allows to define nested sections, lists of items, key-value pairs and more. To
-change a default settings, just define it in the |yaml| as explaned in the
-following sections.
+The following give a description of the contents of |yaml|.
 
 .. note::
 
-   If other settings exist in |yaml|, they will be ignored by |ptx|. This means
-   that you can use |yaml| to store settings for other tools than |ptx|.
+   If other settings not described below exist in |yaml|, they will be ignored
+   by |ptx|. This means that you can use |yaml| to store settings for other
+   purposes than |ptx|.
+
+Runner section
+--------------
 
-Number of parallel processes
-----------------------------
+The purpose of this section is to be able to precisely define how to run the
+|exe| for each test case. The *runner* section contains key-value pairs of
+settings to be used for replacing placeholders in the |runner| passed to
+:option:`--exe-runner`. For a key to be replaced, the |runner| shall contain
+the key between double curly braces.
 
-This parameter is not used by directly by |ptx| for running the |exe|,
-but is intended to be used in the |runner|. To change the number of
-parallel processes:
+For instance, if |yaml| of a test case contains:
 
 .. code-block:: yaml
 
-   nproc: 10
+   runner:
+      nproc: 10
+
+and the |runner| passed to :option:`--exe-runner` contains:
+
+.. code-block:: console
+
+   mpirun -np {{nproc}} executable
 
+then this line in the actual |runner| used to run the test case will be:
 
-Regression reference files
---------------------------
+.. code-block:: console
 
-Reference files are used to do regression checks on the files produced by
-the |exe|. The regression is done by comparing the files with a given
-tolerance (explained in the next section). The `references` setting shall
-contain a list of paths to the files to be compared. A path shall be defined
-relatively to the test case directory, it may use any shell pattern like
-:file:`**`, :file:`*`, :file:`?`, for instance:
+   mpirun -np 10 executable
+
+.. _yaml-ref:
+
+Reference files
+---------------
+
+The reference files are used to check for regressions on the files created by
+the |exe|. Those checks can be done by comparing the files with a tolerance
+, see :ref:`yaml-tol`. The *references* section shall contain a list of paths
+to the files to be compared. A path shall be defined relatively to the test
+case outpput directory, it may use any shell pattern like :file:`**`,
+:file:`*`, :file:`?`, for instance:
 
 .. code-block:: yaml
 
    references:
-      - path/to/file/relative/to/test/case
+      - output/file
+      - '**/*.txt'
 
+Note that |ptx| does not know how to check for regression on files, you have to
+implement the |pytest| tests by yourself. To get the path to the references
+files in a test function, use the fixture :ref:`regression-path-fixtures`.
+
+.. _yaml-tol:
 
 Tolerances
 ----------
 
-To change the tolerance for comparing a quantity in the |exe| outputs to a reference and allow to
-compare a new NewVariable variable:
+A tolerance is used to define how close shall be 2 data to be considered as
+equal. It can be used when checking for regression by comparing files, see
+:ref:`yaml-ref`. To set the tolerances for the data named *data-name1* and
+*data-name2*:
 
 .. code-block:: yaml
 
    tolerances:
-       quantity1:
+       data-name1:
            abs: 1.
-       quantity2:
+       data-name2:
            rel: 0.
            abs: 0.
 
-If one of the tolerance value is not defined, like the **abs** one for the
-**Velocity**, then its value will be set to **0.**.
+For a given name, if one of the tolerance value is not defined, like the
+**rel** one for the **data-name1**, then its value will be set to **0.**.
+
+Note that |ptx| does not know how to use a tolerance, you have to implement it
+by yourself in a |pytest| tests. To get the tolerance in a test function, use
+the :ref:`tolerances-fixtures`.
 
+.. _yaml-marks:
 
 Marks
 -----
 
 A mark is a |pytest| feature that allows to select some of the tests to be
-executed. A mark is a kind of tag or label assigned to a test. This is how to
-add marks to a test case, for instance the **slow** and **big** marks:
+executed, see :ref:`mark_usage`. This is how to add marks to a test case, for
+instance the **slow** and **big** marks:
 
 .. code-block:: yaml
 
    marks:
       - slow
       - big
 
+Such a declared mark will be set to all the test functions under a test case,
+either from the default test module or from an additional |pytest| module.
+
 You can also use the marks that already existing. In particular, the `skip` and
 `xfail` marks provided by |pytest| can be used. The `skip` mark tells pytest to
 record but not execute the built-in test events of a test case. The `xfail`
@@ -134,5 +175,5 @@ is executed. This file shall have the format:
 
    [pytest]
    markers =
-       slow: one line explanation of slow
-       isotropy: one line explanation of isotropy
+       slow: one line explanation of what slow means
+       big: one line explanation of what big means