Skip to content

Commit

Permalink
doc: some improvements
Browse files Browse the repository at this point in the history
  • Loading branch information
Antoine DECHAUME committed May 19, 2020
1 parent 295ac39 commit c2a6d5f
Show file tree
Hide file tree
Showing 4 changed files with 42 additions and 53 deletions.
44 changes: 22 additions & 22 deletions doc/how-to-use.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ The |ptx| plugin can be used in a wide variety of ways, the following sections
explain how.


Run |exe| only
---------------------
Run the |exe| only
------------------

:command:`pytest --exe-runner <path/to/runner> <path/to/tests/inputs> -k runner`

Expand All @@ -40,34 +40,34 @@ more informations on doing only some of the processing.
For instance, if the tests input tree contains::

path/to/tests/inputs
├── dir-1
├── case-1
│ ├── input
│ └── test_case.yaml
└── dir-2
└── case-2
├── input
└── test_case.yaml

Then the tests output tree is::

tests-output
├── dir-1
│ ├── input -> path/to/tests/inputs/dir-1/input
├── case-1
│ ├── input -> path/to/tests/inputs/case-1/input
│ ├── output
│ ├── executable.stderr
│ ├── executable.stdout
│ ├── runner.sh
│ ├── runner.sh.stderr
│ └── runner.sh.stdout
└── dir-2
├── input -> path/to/tests/inputs/dir-2/input
└── case-2
├── input -> path/to/tests/inputs/case-2/input
├── output
├── executable.stderr
├── executable.stdout
├── runner.sh
├── runner.sh.stderr
└── runner.sh.stdout

For a given test case, for instance :file:`tests-output/dir-1`,
For a given test case, for instance :file:`tests-output/case-1`,
the output directory contains:

output
Expand Down Expand Up @@ -101,39 +101,39 @@ runner.sh.stdout

If you need to manually run the |exe| for a test case, for debugging
purposes for instance, just go to its output directory, for instance
:command:`cd tests-output/dir-1`, and execute the |runner|.
:command:`cd tests-output/case-1`, and execute the |runner|.


Do default regression checking without running executable
---------------------------------------------------------
Check regressions without running the |exe|
-------------------------------------------

:command:`pytest --exe-regression-root <path/to/tests/references> <path/to/tests/inputs> --exe-overwrite-output`

We assume that the |exe| results have already been produced for the test cases
considered. This is not enough though because the output directory already
exists and |ptx| will by default prevent the user from silently modifying any
existing test output directories. In that case, the option
:option:`--exe-overwrite-output` shall be used. The above command line will compare
the results in the default output tree with the references, if the existing
|exe| results are in a different directory then you need to add the path to it
with :command:`--exe-output-root`.
:option:`--exe-overwrite-output` shall be used. The above command line will
compare the results in the default output tree with the references, if the
existing |exe| results are in a different directory then you need to add the
path to it with :command:`--exe-output-root`.

The option :option:`--exe-regression-root` points to the root directory with the
regression references tree . This tree shall have the same hierarchy as the
The option :option:`--exe-regression-root` points to the root directory with
the regression references tree . This tree shall have the same hierarchy as the
output tree but it only contains the results files that are used for doing the
regression checks.


Run |exe| and do default regression checks
-------------------------------------------------
Run the |exe| and do default regression checks
----------------------------------------------

:command:`pytest --exe-runner <path/to/runner> --exe-regression-root <path/to/tests/references> <path/to/tests/inputs>`

.. note::

Currently this can only be used when |exe| execution is done on the same
Currently this can only be used when the |exe| execution is done on the same
machine as the one that execute the regression checks, i.e. this will not
work when |exe| is submitted through a job scheduler.
work when the |exe| is submitted through a job scheduler.

Finally, checks are done on the |exe| log files to verify that the file
:file:`executable.stdout` exists and is not empty, and that the file
Expand Down
23 changes: 6 additions & 17 deletions doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,22 +18,15 @@
Welcome to |ptx| documentation!
===============================

This is the user guide for |ptx|, a |pytest| plugin for checking and validating an |exe|.

This is the user guide for |ptx|, a |pytest| plugin for black-box testing an |exe|.

Overview
--------

The |ptx| plugin allows to both automatically check |exe| results and
post-process them. In this guide, a `check` is a testing event that can be
automatically verified and can provide an OK or KO outcome, like checking that
2 numbers are equal. In contrast, a `post-process` is a testing event that
solely produces additional data, like numerical or graphical data, which has to
be analyzed manually in order to be qualified as OK or KO. The |ptx| plugin
may also generate test reports and users may add custom check and
post-processing events.
post-process them.

The |ptx| plugin works with several test cases directory trees for:
The |ptx| plugin deals with multiple directory trees:

- the inputs
- the outputs
Expand All @@ -44,13 +37,9 @@ sets of references, for instance for comparing the results against more than
one version of |exe|. All the directory trees have the same hierarchy,
this convention allows |ptx| to work out what to test and what to check.
Except for the inputs tree, you do not have to manually create the directory
hierarchies, as they are automatically created by |ptx| when it is executed.

In the inputs tree, a test case is a directory that contains:
hierarchies, as they are automatically created by |ptx|.

- the |exe| input files
- a |yaml| file with basic settings
- optionnal |pytest| and python scripts for adding checks and post-processes
To create a test case, see :ref:`add-test-case-label`.

In the outputs tree, a test case directory typically contains:

Expand All @@ -59,7 +48,7 @@ In the outputs tree, a test case directory typically contains:
- the files produced by the execution of |exe|
- eventually, the files produced by the additional post-processing

In a regression references tree, a test case directory shall contains all the
In a regression references tree, a test case directory shall contain all the
result files required for performing the checks.


Expand Down
22 changes: 10 additions & 12 deletions doc/test-case.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ A test case is composed of an input directory with:

.. warning::

An test case input directory shall not contain any of the files created by
The input directory of a test case shall not contain any of the files created by
the execution of the |exe| or of the additional python modules, otherwise
they may badly interfere with the executions done by |ptx|. In other words:
do not run anything in the input directory of a test case, this directory
Expand All @@ -37,24 +37,23 @@ A test case is composed of an input directory with:
The |yaml| file is used by |ptx| for several things. When this file is
found, |ptx| will:

1. create the test case output directory and, if needed, its parents,
2. read the settings for running the test case,
3. go to the test case output directory and execute the tests defined
4. in the default test module,
5. then in the additional test modules.
1. create the output directory of the test case and, if needed, its parents,
4. execute the tests defined in the default test module,
5. execute the tests defined in the additional test modules.
5. execute the tests defined in the parent directories.

The parents of the output directory are created such that the path from the
directory where |pytest| is executed to the test case input directory is the
same but for the first parent. This way, the directories hierarchy below the
first parent of both the inputs and the outputs trees are the same.
The parents of an output directory are created such that the path from the
directory where |pytest| is executed to the input directory of the test case is
the same but for the first parent. This way, the directories hierarchy below
the first parent of both the inputs and the outputs trees are the same.

If |yaml| is empty, then the default settings are used. If
:option:`--exe-default-settings` is not set, the default settings are the
builtin ones:

.. literalinclude:: ../src/pytest_executable/test_case.yaml

The following give a description of the contents of |yaml|.
The following gives a description of the contents of |yaml|.

.. note::

Expand Down Expand Up @@ -162,7 +161,6 @@ record but not execute the built-in test events of a test case. The `xfail`
mark tells pytest to expect that at least one of the built-in test events will
fail.


Marks declaration
-----------------

Expand Down
6 changes: 4 additions & 2 deletions doc/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
.. _conda: https://docs.conda.io
.. _pip: https://pip-installer.org
.. _report-conf: https://github.com/CS-SI/pytest-executable/tree/master/report-conf


Installation
Expand Down Expand Up @@ -94,12 +95,13 @@ Plugin options

use PATH as the script to generate the test report

See the :file:`report-conf` directory for an example of such a script.
See :file:`generate_report.py` in the `report-conf`_ directory for an
example of such a script.

.. note::

The report generator script may require to install additionnal
dependencies, such as sphinx, which are not required by the plugin.
dependencies, such as sphinx, which are not install by the |ptx| plugin.


.. _filter:
Expand Down

0 comments on commit c2a6d5f

Please sign in to comment.