Test Management

Introduction / Overview

These slides give an excellent overview about possible solutions to test Modelica libraries in a good manner.

Unit Tests

We want to make sure that all models in the library work correctly and that changes do not introduce unwanted side effects to exiting models or break them. Instead of testing each model manually, we use the unit testing functions of the Python package BuildingsPy to automate the process. This process is very important for quality control of our library development. AixLib unit testing bases on the IBPSA core library's unit test recommendations. Read this wiki page first.

A unit test for a Modelica model in our case consists of an example model that is automatically simulated by a *.mos-Script and a Python routine that compares the simulation results of the example model with stored reference results. Thus, if something causes the example model's results to change, this can be automatically detected.

Setting up your system for running the unit tests

This section assumes you are on a Windows system and have Dymola and Python installed. As BuildingsPy currently only supports Python 2, the Python installation should be a recent version of Python 2.

In order to run the unit tests of AixLib you will need a few things:

Have BuildingsPy installed

When installing BuildingsPy (and ModelicaRes), we recommend to clone their respective repositories. You should then have a local version of them at <path-to-your-cloned-repository>.Then install the packages by typing pip install -e <path-to-your-cloned-repository> into your cmd prompt. This is because other AixLib developers will also work with the current master of these packages from the repositories instead of with current official release versions. Therefore, using pip install buildingspy or pip install modelicares may lead to you using different versions than other AixLib developers.

Important: Currently for the coverage determination we have a line of code in our runUnitTests.py which is not supported by the original Buildingspy library. Please comment out this line for running unittests.

Have pytidylib installed and libtidy.dll set up

pytidylib should automatically be collected with the installation of the BuildingsPy package. Otherwise, you can install it with pip install pytidylib. Unfortunately, on Windows you also have to make sure to set up libtidy.dll. To do this, follow these steps:

• Download the libtidy.dll from http://www.paehl.com/open_source/?HTML_Tidy_for_Windows (Click on LIBTIDY DLL)
• Un-zip the file to a directory of your choice (let's call the path <path-to-your-libtidy.dll>)
• Set your PATH variable to point to the libtidy.dll-file, i.e. add <path-to-your-libtidy.dll> to your PATH variable (Info from http://countergram.com/open-source/pytidylib/docs/index.html)

Have matplotlib and numpy installed

As fundamental changes to these packages happen more rarely, it is ok to install them by simply typing pip install matplotlib and pip install numpy into your cmd prompt. That is just in case you do not have them already installed, which is the default for many Python distributions. In that case, it does not hurt to update to the latest release with pip install numpy --upgrade or pip install matplotlib --upgrade .

Have your PATH environmental variable point to your Dymola executable

In order to be able to start Dymola for the simulation of the unit test example model, PATH must point to the Dymola executable. Therefore, add e.g. C:\Program Files\Dymola 2020\bin64 to your PATH variable.

With all this in place, you should be ready to go and be able to run the unit tests.

Run the unit tests

In order to run the unit tests open a command prompt and cd to the directory into which you cloned the AixLib repository. Let's call this directory <path-to-AixLib>. In <path-to-AixLib> there should be the directory AixLib that contains the package.mo file for the library. Next, cd to AixLib, so that your command prompt is now at <path-to-AixLib>\AixLib. From here, you can run the unit tests by typing ..\bin\runUnitTests.py.

As a default, the unit tests simulate the example models and compare the simulation results with the reference results. In addition, the unit tests check the model documentations for correct html syntax. In some cases, it may be useful not to run all functions of the unit tests, but only part of it. For this, the unit test script offers a few modifiers:

Run unit tests to validate html syntax only, without simulating models

When updating the documentation of models, it may be faster to only run the html syntax check on the models, without simulating all the models. To do this, run the unit tests with the modifier --validate-html-only. Thus, from <path-to-AixLib>\AixLib execute ..\bin\runUnitTests.py --validate-html-only.

Run unit tests in batch mode, without user interaction

In normal operation, the unit tests will require a user interaction if any changes from the reference results are detected. To avoid this user interaction, use the modifier -b or --batch. Thus, from <path-to-AixLib>\AixLib execute ..\bin\runUnitTests.py -b.

Specify the number of processors available for the unit tests

As default, the unit tests will use all available processors. If you want to limit the number of processors used for the unit tests, use the modifier -n <number-of-processors>. Thus, for example for using only 2 processors, from <path-to-AixLib>\AixLib execute ..\bin\runUnitTests.py -n 2.

Run unit tests for a subpackage only, not for all of AixLib

Especially for development, it may be useful to only run the unit tests for a specific subpackage, instead of for the whole library. To do this, use the modifier -s <AixLib.package>, with <AixLib.package> being a model path in dot notation. E.g. to only test the low order building models, from <path-to-AixLib>\AixLib execute ..\bin\runUnitTests.py -s AixLib.Building.LowOrder. This will test the specified package and all its subpackages.

Maximal example

Use
python "C:\Path\to\lib\AixLib\bin\runUnitTests.py --help
to get all available options.

A two liner could look like (Windows cmd):

cd "C:\Path\to\lib\AixLib\AixLib"
python ../bin/runUnitTests.py --batch --single-package AixLib.ThermalZones.ReducedOrder --path "C:/Path/to/lib/AixLib/AixLib" --number-of-processors 3 --show-gui --skip-verification

Be aware that --batch overwrites the options --show-gui and --skip-verification.

Write a unit test

1. Chose an example that does not have a unit test(maybe the example you just implemented).
2. Make sure the example translates without warnings.
3. Set up a simulation, fulfilling the requirements from https://github.com/ibpsa/modelica-ibpsa/wiki/Unit-Tests ("How to include models as part of the unit tests", points 1-6)
   • I recommend using the number of intervals (500) for the output, instead of an interval length.
   • "Store in model" the simulation set up in order to get the "experiment" annotation.
4. Make sure the simulation runs without errors or warnings.
5. Chose the relevant variables to plot and plot them in a way you like.
6. Generate a script for the plot (Simulation Window -> File -> Generate script -> Store script for plot setup). Save this script with the name of the example, this will be the script for your unit test.
7. Add your simulation command to this script as the first command (copy/paste from the Dymola command line).
8. Save this script in the corresponding sub-directories under "AixLib\Resources\Scripts\Dymola". The structure of the sub-directories has to mirror the library package hierarchy.
9. Add a command for running this script (Simulation Window -> Commands -> Add command -> Run script + Description 'Simulate and plot')
10. Run the BuildingsPy routine runUnitTests.py for your example to check that everything runs correctly and generate reference results
    • Make sure you satisfy the "Software requirements" listed from https://github.com/ibpsa/modelica-ibpsa/wiki/Unit-Tests
    • Go into the folder where you have AixLib (all the packages).
    • Run the following command python ../bin/runUnitTests.py --single-package PackagePathInAixLib --tool dymola. This will run for the whole package where your example is located.
    • When asked to save the reference results say yes.
    • Solve any other warnings which might appear.
11. Commit your script and reference results.

New Reference Results

You should accept new results for your new unit test and commit these results before submitting a pull request. If results have changed for existing unit tests, only accept the changes if you can explain them and make sure to document the reasons for the change on the issue tracker / in the pull request.