-
Notifications
You must be signed in to change notification settings - Fork 91
Backward Compatibility Testing
A script is available for backward compatibility testing of the SkillMetrics package when implementing updates or new capabilities. This is done to ensure code updates retain backward compatibility of the diagrams created, and to safeguard the occurrence of bugs when new capabilities are implemented. This is accomplished by comparing the images in the plots created by the example scripts against the reference ones posted on GitHub, e.g. target1.png with respect to target1_example.png.
The test_plots.py script facilitates identification of undesired side effects of updates related to the generation of target and Taylor diagrams using the existing examples.
It supports the following arguments as options.
-clean_files : Delete new files if this flag is present
-h : help
It can be invoked from a command line as, for example, to remove the new files:
$ python test_plots.py -clean_files
The python scripts executed are those assigned to the SCRIPTS_TESTED constant in test_plots.py and that reside in the Examples folder. These scripts are executed without displaying the plots but write new example graphics files.
The new files are compared pixel-wise with the previously existing example graphic files and a percentage change is displayed for each example.
Percentage change: 100% * sum of pixel color differences / mean colors intensity
Some image files are generated with different width/height depending on the system. When the size of the current image and the size of the new image are different, both are resized to have the same size and then compared. This is reported in the script output if it occurs.
Changes less than 2% may be considered negligible (GOOD). Between 2% and 10% should be double checked (CHECK). More than 10% is a probable case of a difference arising due to a bug (BAD). Note that the resized images are expected to have bigger 'distances' due to changes caused by the resizing step, and that these percentage thresholds were taken arbitrarily. The test_plot script reports these outcomes according to the percentage.
The list of scripts to be processed must be assigned to SCRIPTS_TESTED as a list of tuples, and can make use of regular expressions (regex) for filename pattern matching that is supported by the glob module. For example, to process all
target and taylor scripts:
SCRIPTS_TESTED = ("target*[0-9].py", "taylor*[0-9].py")
To process only the target or taylor scripts:
SCRIPTS_TESTED = ("target*[0-9].py", ) # target diagrams only, must be a tuple
SCRIPTS_TESTED = ("taylor*[0-9].py", ) # Taylor diagrams only, must be a tuple
An example of the script output for only example target diagrams is shown below. The list of files are processed in sorted alphabetical order.
$ python test_plots.py
Executing 9 scripts in sequence:
01: 0.86% distance between 'target1_example.png' and 'target1.png', GOOD.
02: 0.15% distance between 'target2_example.png' and 'target2.png', GOOD.
03: 0.15% distance between 'target3_example.png' and 'target3.png', GOOD.
04: 0.00% distance between 'target4_example.png' and 'target4.png', GOOD.
05: 0.00% distance between 'target5_example.png' and 'target5.png', GOOD.
06: 0.11% distance between 'target6_example.png' and 'target6.png', GOOD.
07: 0.00% distance between 'target7_example.png' and 'target7.png', GOOD.
08: 2.64% distance between 'target8_example.png' and 'target8.png', CHECK (size adjusted).
09: 2.48% distance between 'target9_example.png' and 'target9.png', CHECK (size adjusted).
New files were kept after comparison.
It is expected that test cases will be added over time as the SkillMetrics package continues to evolve.
A zip file containing the scripts and PNG files needed to execute backward compatibility testing is available here. It can be found in the Test folder and is provided for ease of download.
A heartfelt thank you is given to the author for this time saving contribution.
Andre D. L. Zanchetta ("adlzanchetta" in multiple social media), adlzanchetta@gmail.com
Backward Compatibility Test Script: test_plots.py