Skip to content

Commit

Permalink
feat: change cli option names to avoid clashes with other plugins
Browse files Browse the repository at this point in the history
  • Loading branch information
Antoine DECHAUME committed May 10, 2020
1 parent 62548f4 commit d438913
Show file tree
Hide file tree
Showing 9 changed files with 93 additions and 75 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Changed
~~~~~~~
- The name of the runner shell script in the output directories is the one
passed to the CLI instead of the hardcoded one.
- All the names of the CLI options have been prefixed with *exe* to prevent
from clashing with other plugins CLI names.

Added
~~~~~
Expand Down
4 changes: 2 additions & 2 deletions doc/fixtures.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Runner fixture
This fixture is used to run |exe|, it will do the following:

- get the runner script passed to the |pytest| command line option
:option:`--runner`,
:option:`--exe-runner`,
- process it to replace the placeholders `{{nproc}}` and `{{output_path}}` with their
actual values,
- write it to the |runner| in the test case output directory.
Expand Down Expand Up @@ -71,7 +71,7 @@ Regression path fixture

This fixture is used to get the absolute path to the directory that contains
the regression reference of a test case when the command line option
:option:`--regression-root` is used. It provides the :py:data:`regression_path`
:option:`--exe-regression-root` is used. It provides the :py:data:`regression_path`
variable that holds a `Path`_ object.

You may use this fixture with the :py:data:`output_path` fixture to get the
Expand Down
14 changes: 7 additions & 7 deletions doc/how-to-use.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@ explain how.
Run |exe| only
---------------------

:command:`pytest --runner <path/to/runner> <path/to/tests/inputs> -k runner`
:command:`pytest --exe-runner <path/to/runner> <path/to/tests/inputs> -k runner`

This command will execute the |exe| for all the test cases that are found in
the input tree under :file:`path/to/tests/inputs`. A test case is identified by
a directory that contains a |yaml| file. For each of the test cases found,
|ptx| will create an output directory with the same directory hierarchy and run
the cases in that output directory. By default, the root directory of the
output tree is :file:`tests-output`, this can be changed with the option
:option:`--output-root`. Finally, the :option:`-k runner` option instructs
:option:`--exe-output-root`. Finally, the :option:`-k runner` option instructs
|pytest| to only execute the |runner| and nothing more, see :ref:`filter` for
more informations on doing only some of the processing.

Expand Down Expand Up @@ -105,18 +105,18 @@ purposes for instance, just go to its output directory, for instance
Do default regression checking without running executable
---------------------------------------------------------

:command:`pytest --regression-root <path/to/tests/references> <path/to/tests/inputs> --overwrite-output`
:command:`pytest --exe-regression-root <path/to/tests/references> <path/to/tests/inputs> --exe-overwrite-output`

We assume that the |exe| results have already been produced for the test cases
considered. This is not enough though because the output directory already
exists and |ptx| will by default prevent the user from silently modifying any
existing test output directories. In that case, the option
:option:`--overwrite-output` shall be used. The above command line will compare
:option:`--exe-overwrite-output` shall be used. The above command line will compare
the results in the default output tree with the references, if the existing
|exe| results are in a different directory then you need to add the path to it
with :command:`--output-root`.
with :command:`--exe-output-root`.

The option :option:`--regression-root` points to the root directory with the
The option :option:`--exe-regression-root` points to the root directory with the
regression references tree . This tree shall have the same hierarchy as the
output tree but it only contains the results files that are used for doing the
regression checks.
Expand All @@ -125,7 +125,7 @@ regression checks.
Run |exe| and do default regression checks
-------------------------------------------------

:command:`pytest --runner <path/to/runner> --regression-root <path/to/tests/references> <path/to/tests/inputs>`
:command:`pytest --exe-runner <path/to/runner> --exe-regression-root <path/to/tests/references> <path/to/tests/inputs>`

.. note::

Expand Down
18 changes: 9 additions & 9 deletions doc/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ root directory.
Plugin options
--------------

.. option:: --runner PATH
.. option:: --exe-runner PATH

use the shell script at PATH to run |exe|, if omitted then |exe| is
not run.
Expand All @@ -55,30 +55,30 @@ Plugin options
.. literalinclude:: ../mpi_runner.sh
:language: bash

.. option:: --output-root PATH
.. option:: --exe-output-root PATH

use PATH as the root for the output directory tree, default: tests-output

.. option:: --overwrite-output
.. option:: --exe-overwrite-output

overwrite existing files in the output directories

.. option:: --clean-output
.. option:: --exe-clean-output

clean the output directories before executing the tests

.. option:: --regression-root PATH
.. option:: --exe-regression-root PATH

use PATH as the root directory with the references for the regression
testing, if omitted then the tests using the regression_path fixture will be
skipped

.. option:: --default-settings PATH
.. option:: --exe-default-settings PATH

use PATH as the yaml file with the global default test settings instead of
the built-in ones

.. option:: --report-generator PATH
.. option:: --exe-report-generator PATH

use PATH as the script to generate the test report

Expand All @@ -104,11 +104,11 @@ Use multiple path patterns
Instead of providing the path to the root of the inputs tree, you may
provide the path to one or more of its sub-directories, for instance:

:command:`pytest --runner <path/to/runner> <path/to/tests/inputs/sub-directory1> <path/to/tests/inputs/sub/sub/sub-directory2>`
:command:`pytest --exe-runner <path/to/runner> <path/to/tests/inputs/sub-directory1> <path/to/tests/inputs/sub/sub/sub-directory2>`

You may also use shell patterns (with `*` and `?` characters) in the paths like:

:command:`pytest --runner <path/to/runner> <path/to/tests/inputs/*/sub-directory?>`
:command:`pytest --exe-runner <path/to/runner> <path/to/tests/inputs/*/sub-directory?>`

Use marks
A test case could be assigned one or more marks in the |yaml| file, then
Expand Down
66 changes: 37 additions & 29 deletions src/pytest_executable/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,44 +50,50 @@
def pytest_addoption(parser):
"""CLI options for the plugin."""
group = parser.getgroup("executable", "executable testing")

group.addoption(
"--runner",
"--exe-runner",
metavar="PATH",
help="use the shell script at PATH to run an executable, if omitted then "
"the executable is not run but the other test processing will be",
)

group.addoption(
"--output-root",
"--exe-output-root",
default="tests-output",
metavar="PATH",
help="use PATH as the root directory of the tests output, default: %(default)s",
)

group.addoption(
"--overwrite-output",
"--exe-overwrite-output",
action="store_true",
help="overwrite existing files in the tests output directories",
)

group.addoption(
"--clean-output",
"--exe-clean-output",
action="store_true",
help="clean the tests output directories before executing the tests",
)

group.addoption(
"--regression-root",
"--exe-regression-root",
metavar="PATH",
help="use PATH as the root directory with the references for the "
"regression testing, if omitted then the tests using the regression_path "
"fixture will be skipped",
)

group.addoption(
"--default-settings",
"--exe-default-settings",
default=SETTINGS_PATH,
metavar="PATH",
help="use PATH as the yaml file with the global default test settings instead "
"of the built-in ones",
)
group.addoption(
"--report-generator",
"--exe-report-generator",
metavar="PATH",
help="use PATH as the script to generate the test report",
)
Expand All @@ -106,16 +112,16 @@ def pytest_sessionstart(session):
getoption = session.config.getoption

# check options clash
if getoption("clean_output") and getoption("overwrite_output"):
msg = "options --clean-output and --overwrite-output are not compatible"
if getoption("exe_clean_output") and getoption("exe_overwrite_output"):
msg = "options --exe-clean-output and --exe-overwrite-output are not compatible"
raise pytest.UsageError(msg)

# check paths are valid
for option_name in (
"runner",
"default_settings",
"regression_root",
"report_generator",
"exe_runner",
"exe_default_settings",
"exe_regression_root",
"exe_report_generator",
):
path = getoption(option_name)
try:
Expand Down Expand Up @@ -147,31 +153,31 @@ def _get_parent_path(fspath: py.path.local) -> Path:
def create_output_tree(request):
"""Fixture to create and return the path to the output directory tree."""
getoption = request.config.getoption
output_root = Path(getoption("output_root"))
output_root = Path(getoption("exe_output_root"))
parent_path = _get_parent_path(request.node.fspath)
output_path = get_mirror_path(parent_path, output_root.resolve())

try:
create_output_directory(
parent_path,
output_path,
not getoption("overwrite_output"),
getoption("clean_output"),
not getoption("exe_overwrite_output"),
getoption("exe_clean_output"),
OUTPUT_IGNORED_FILES,
)
except FileExistsError:
msg = (
f'the output directory "{output_path}" already exists: either '
"remove it manually or use the --clean-output option to remove "
"it or use the --overwrite-output to overwrite it"
"remove it manually or use the --exe-clean-output option to remove "
"it or use the --exe-overwrite-output to overwrite it"
)
raise FileExistsError(msg)


@pytest.fixture(scope="module")
def output_path(request):
"""Fixture to return the path to the output directory."""
output_root = Path(request.config.getoption("output_root")).resolve(True)
output_root = Path(request.config.getoption("exe_output_root")).resolve(True)
return get_mirror_path(_get_parent_path(request.node.fspath), output_root)


Expand All @@ -186,7 +192,7 @@ def _get_settings(config: _pytest.config.Config, path: Path) -> Settings:
The settings from the test case yaml.
"""
return Settings.from_local_file(
Path(config.getoption("default_settings")),
Path(config.getoption("exe_default_settings")),
_get_parent_path(path) / SETTINGS_PATH.name,
)

Expand All @@ -203,17 +209,17 @@ def runner(request, create_output_tree, output_path):
This fixture will create an executable runner script in the test case
output directory from the script passed to the pytest command line with the
option :option:`--runner`. The placeholders {nproc} and {output_path} are
option :option:`--exe-runner`. The placeholders {nproc} and {output_path} are
replaced with their actual values in the written script. The runner object
created by the fixture can be executed with the :py:meth:`run` method which
will return the return code of the script execution.
Returns:
ScriptRunner object.
"""
runner_path = request.config.getoption("runner")
runner_path = request.config.getoption("exe_runner")
if runner_path is None:
pytest.skip("no runner provided to --runner")
pytest.skip("no runner provided to --exe-runner")

# check path
runner_path = Path(runner_path).resolve(True)
Expand All @@ -230,7 +236,7 @@ def _get_regression_path(
) -> Optional[Path]:
"""Return the path to the reference directory of a test case.
None is returned if --regression-root is not passed to the CLI.
None is returned if --exe-regression-root is not passed to the CLI.
Args:
config: Config from pytest.
Expand All @@ -239,7 +245,7 @@ def _get_regression_path(
Returns:
The path to the reference directory of the test case or None.
"""
regression_path = config.getoption("regression_root")
regression_path = config.getoption("exe_regression_root")
if regression_path is None:
return None
return get_mirror_path(
Expand All @@ -252,7 +258,9 @@ def regression_path(request):
"""Fixture to return the path of a test case under the references tree."""
regression_path = _get_regression_path(request.config, request.node.fspath)
if regression_path is None:
pytest.skip("no tests references root directory provided to --regression-root")
pytest.skip(
"no tests references root directory provided to --exe-regression-root"
)
return regression_path


Expand All @@ -261,7 +269,7 @@ def pytest_generate_tests(metafunc):
Used for accessing the references files.
If --regression-root is not set then no reference files will be provided.
If --exe-regression-root is not set then no reference files will be provided.
"""
if "regression_file_path" not in metafunc.fixturenames:
return
Expand Down Expand Up @@ -412,15 +420,15 @@ def pytest_terminal_summary(
created and the report generator is called.
"""
# path to the report generator
reporter_path = config.getoption("report_generator")
reporter_path = config.getoption("exe_report_generator")
if reporter_path is None:
return

if not terminalreporter.stats:
# no test have been run thus no report to create or update
return

output_root = Path(config.getoption("output_root"))
output_root = Path(config.getoption("exe_output_root"))

terminalreporter.write_sep("=", "starting report generation")

Expand Down
2 changes: 1 addition & 1 deletion src/pytest_executable/script_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
import stat
import subprocess
from pathlib import Path
from typing import Any, Dict, Iterator
from typing import Any, Dict

import jinja2

Expand Down

0 comments on commit d438913

Please sign in to comment.