Skip to content

Commit

Permalink
Merge pull request #21 from pstelzig/19-include-testgen-script-in-rel…
Browse files Browse the repository at this point in the history
…ease

19 include testgen script in release
  • Loading branch information
pstelzig committed Oct 27, 2023
2 parents a136d13 + f019cbb commit b89de92
Show file tree
Hide file tree
Showing 6 changed files with 58 additions and 37 deletions.
15 changes: 12 additions & 3 deletions .github/workflows/job-examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,19 @@ jobs:
run: |
cd examples
python3 -m unittest
- name: Run test generation examples
- name: Run test generation examples with command line tool
run: |
cd examples/generate_tests
/github/home/.local/bin/mopyregtest ./gen_tests BlocksLpDist_from_cli \
~/".openmodelica/libraries/Modelica 4.0.0+maint.om/" \
Modelica.Blocks.Sources.Sine,Modelica.Blocks.Sources.ExpSine,Modelica.Blocks.Sources.Step \
--metric=Lp_dist
cd gen_tests
python3 test_blockslpdist_from_cli.py
- name: Run test generation examples with script
run: |
cd examples/generate_tests
python3 gentests_modelica_blocks_sources.py
cd gen_tests
python3 test_blockslpdist.py
python3 test_blocksuserdef.py
python3 test_blockslpdist_from_script.py
python3 test_blocksuserdef_from_script.py
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MoPyRegtest: A Python enabled simple regression testing framework for Modelica models

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](/LICENSE)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/pstelzig/MoPyRegtest/tree/master/LICENSE)
[![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/pstelzig/MoPyRegtest/issues)

Scope of this project is to
Expand Down Expand Up @@ -47,15 +47,15 @@ The user has to
* create a Python file and test methods that instantiate a `mopyregtest.RegressionTest` object,
which needs to know which Modelica model to test and where to find the reference result.

Examples can be found in the `examples` folder. Also see the [examples/README.md](/examples/README.md).
Examples can be found in the `examples` folder. Also see the [examples/README.md](https://github.com/pstelzig/MoPyRegtest/tree/master/examples/README.md).

### Automatic test case generation
MoPyRegtest can generate regression test definitions for elements of a Modelica library automatically.
Either by creating [Generator class objects](/mopyregtest/generator.py). Or from the command line using the
[testgen.py script](/tools/testgen.py).
Either by creating [Generator class objects](https://github.com/pstelzig/MoPyRegtest/tree/master/mopyregtest/generator.py). Or using the
[`mopyregtest` command line tool](https://github.com/pstelzig/MoPyRegtest/blob/master/mopyregtest/cli.py).

The [examples folder generate_tests](/examples/generate_tests) has a detailed example on how to automatically
generate tests using a Python script.
The folder [examples/generate_tests/README.md](https://github.com/pstelzig/MoPyRegtest/tree/master/examples/generate_tests/README.md)
has a detailed explanation on how to automatically generate tests.

## Prerequisites
To use `MoPyRegtest` you need to have
Expand All @@ -74,7 +74,7 @@ You can install `MoPyRegtest` from the [Python Package Index](https://pypi.org/p
pip3 install --user mopyregtest
```

To install a specific version, e.g. `v0.3.1`, run `pip3 install --user mopyregtest==0.3.1`
To install a specific version, e.g. `v0.3.2`, run `pip3 install --user mopyregtest==0.3.2`

### Install from source
To install from source, first clone this repository to your local `<your-mopyregtest-dir>` (adapt to your needs) with
Expand All @@ -87,11 +87,11 @@ cd <your-mopyregtest-dir>
pip3 install --user .
```

to get the latest development head. To install a specific version from source, e.g. `v0.3.1`, run
to get the latest development head. To install a specific version from source, e.g. `v0.3.2`, run

```bash
cd <your-mopyregtest-dir>
git checkout v0.3.1
git checkout v0.3.2
pip3 install --user .
```

Expand Down
32 changes: 19 additions & 13 deletions examples/generate_tests/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Automatic test definition example
In [gentests_modelica_blocks_sources.py](/examples/generate_tests/gentests_modelica_blocks_sources.py) you find a
complete example on how automatic test generation works using a Python script. The script basically generates code
complete example on how automatic test generation works in `MoPyRegtest`. The script basically generates code
like you would create manually like in the example
[test_Modelica_Electrical_Analog_Examples](/examples/test_Modelica_Electrical_Analog_Examples).
Below you will find a detailed explanation of the generation example script.

Tests can also be generated automatically from the command line using the [testgen.py tool](/tools/testgen.py).
Tests can also be generated automatically from the command line using the `mopyregtest` command line tool.

Make sure to have `MoPyRegtest` installed via `pip`, see [examples/README.md](/examples/README.md).

Expand All @@ -14,17 +14,19 @@ First, we will generate the same test that is generated by the script in
[gentests_modelica_blocks_sources.py](/examples/generate_tests/gentests_modelica_blocks_sources.py),
but using the command line.

To see how the command line tool [testgen.py](/tools/testgen.py) works, run
On installation with `pip`, a command line tool called `mopyregtest` is created. As the [setup.py](/setup.py) states in the
`console_scripts` option, this command line tool is a wrapper around [`main` function of cli.py](/mopyregtest/cli.py).
To see how it works, open a command line terminal and run

```bash
cd examples/generate_tests
python3 ../../tools/testgen.py --help
mopyregtest --help
```

which will print

```bash
usage: testgen.py [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--references REFERENCES]
test_folder test_name package_folder models_in_package
usage: mopyregtest [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--references REFERENCES]
test_folder test_name package_folder models_in_package

positional arguments:
test_folder Path where test shall be generated. Advice: Should not exist yet
Expand All @@ -35,11 +37,14 @@ positional arguments:
options:
-h, --help show this help message and exit
--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}
Metric to be used. Choose here from predefined values. For user-defined metrics please consider creating the tests with a dedicated
script.
Metric to be used. Choose here from predefined values. For user-defined metrics please consider creating the tests with a
dedicated script.
--references REFERENCES
Comma separated list like <model name1>:</path/to/ref1.csv>,<model name2>:</path/to/ref2.csv>. Missing references for models here will
be generated.
Comma separated list like <model name1>:</path/to/ref1.csv>,<model name2>:</path/to/ref2.csv>. Missing references for models
here will be generated.

Command line interface for the experimental MoPyRegtest test case generator. This interface is a simplified version. If you want to use all options,
please consider creating a dedicated Python script.
```
Just like in the
Expand All @@ -50,14 +55,15 @@ examples/generate_tests/gen_tests. The test shall be named `BlocksLpDist`. To ac
change to the folder [examples/generate_tests](/examples/generate_tests), and simply run
```bash
python3 ../../tools/testgen.py ./gen_tests BlocksLpDist ~/".openmodelica/libraries/Modelica 4.0.0+maint.om/" Modelica.Blocks.Sources.Sine,Modelica.Blocks.Sources.ExpSine,Modelica.Blocks.Sources.Step --metric=Lp_dist
cd examples/generate_tests
mopyregtest ./gen_tests BlocksLpDist_from_cli ~/".openmodelica/libraries/Modelica 4.0.0+maint.om/" Modelica.Blocks.Sources.Sine,Modelica.Blocks.Sources.ExpSine,Modelica.Blocks.Sources.Step --metric=Lp_dist
```
and for Windows just adapt the path to the Modelica Standard library, e.g. to
`C:/Users/<your user name>/AppData/Roaming/.openmodelica/libraries/Modelica 4.0.0+maint.om`.
You will find the resulting test definition in gen_tests/test_blockslpdist.py and the generated reference results in
gen_tests/references. Note that you can also pass existing reference results to [testgen.py](/tools/testgen.py)
gen_tests/references. Note that you can also pass existing reference results to the `mopyregtest` command line tool
through the `--references` option. E.g., if `Modelica.Blocks.Sources.ExpSine` had an existing reference result in
`/home/<your user name>/Modelica.Blocks.Sources.ExpSine_res.csv`, then you would have to add the option
Expand Down
4 changes: 2 additions & 2 deletions examples/generate_tests/gentests_modelica_blocks_sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@

gen = mopyregtest.Generator(package_folder=package_folder, models_in_package=models_in_package,
metric=mopyregtest.metrics.Lp_dist)
gen.generate_tests(test_folder=this_folder / "gen_tests", test_name="BlocksLpDist",
gen.generate_tests(test_folder=this_folder / "gen_tests", test_name="BlocksLpDist_from_script",
test_results_folder=this_folder / "results")

# Generating a test battery with a user defined metric #################################################################
Expand All @@ -50,5 +50,5 @@

gen = mopyregtest.Generator(package_folder=package_folder, models_in_package=models_in_package,
metric="lambda r_ref, r_act: np.linalg.norm(r_ref[:, 1] - r_act[:, 1], ord=np.inf)")
gen.generate_tests(test_folder=this_folder / "gen_tests", test_name="BlocksUserDef",
gen.generate_tests(test_folder=this_folder / "gen_tests", test_name="BlocksUserDef_from_script",
test_results_folder=this_folder / "results")
19 changes: 10 additions & 9 deletions tools/testgen.py → mopyregtest/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@

import argparse
import pathlib
import mopyregtest
from mopyregtest import metrics
from mopyregtest import Generator

if __name__ == "__main__":

def main():
parser = argparse.ArgumentParser(
epilog="Command line interface for the experimental MoPyRegtest test case generator. "
"This interface is a simplified version. If you want to use all options, please consider "
Expand All @@ -33,7 +35,6 @@

test_name = args.test_name
test_folder = pathlib.Path(args.test_folder)
reference_folder = "references"
result_folder = "results"
package_folder = args.package_folder
models_in_package = args.models_in_package.split(",")
Expand All @@ -43,18 +44,18 @@
else:
references = None

metric = mopyregtest.metrics.norm_infty_dist
metric = metrics.norm_infty_dist
if args.metric is not None:
if args.metric == "norm_p_dist":
metric = mopyregtest.metrics.norm_p_dist
metric = metrics.norm_p_dist
elif args.metric == "norm_infty_dist":
metric = mopyregtest.metrics.norm_infty_dist
metric = metrics.norm_infty_dist
elif args.metric == "Lp_dist":
metric = mopyregtest.metrics.Lp_dist
metric = metrics.Lp_dist
elif args.metric == "Linfty_dist":
metric = mopyregtest.metrics.Linfty_dist
metric = metrics.Linfty_dist
else:
raise ValueError("Invalid value for metric")

gen = mopyregtest.Generator(package_folder=package_folder, models_in_package=models_in_package, metric=metric)
gen = Generator(package_folder=package_folder, models_in_package=models_in_package, metric=metric)
gen.generate_tests(test_folder, test_name, result_folder, references)
7 changes: 6 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,10 @@
install_requires=[
"numpy",
"pandas"
]
],
entry_points={
'console_scripts': [
'mopyregtest = mopyregtest.cli:main',
]
}
)

0 comments on commit b89de92

Please sign in to comment.