Program Analysis Testing

Testing Suites

Here we outline the general structure of the newly created (Spring 2022) testing suites. All of these testing suite use pytest's parameterize functionality to automatically create tests based on test data. The structure of each test is roughly as follows: Each test file defines a constant TEST_DATA_DIR where all test data is stored.

• Each test uses two files for test data
  1. An input file from the previous step of the PA pipeline
  2. the expected result file for the step of the pipeline being tested
• Tests for the above pairs of files are dynamically generated using pytest's parameterize functionality based on a test names. To make this work correctly, the expectation is that the input file and expected result file will follow an expected pattern. In each test file we specify the exact pattern, but in general it is something like name_of_test_{input_file_suffix} and name_of_test_{expected_file_suffix}. To add new tests to a suite of tests, one must simply add the input file and expected result to TEST_DATA_DIR and the testing suite will automatically collect it as a new test case.

GCC Plugin

Test Python file: automatates/tests/program_analysis/GCC2GrFN/test_gcc_plugin_c.py This suite runs the GCC plugin to create a gcc_ast.json file, and compares the created file from an expected json file. To perform the comparison, we the SimpleNamespace from Python's types module. Basically, SimpleNamespace allows us to use dot . access on elements of the json instead of dictionary like access. In the test file, we define functions to compare the generated gcc json with the expected gcc json. These comparison functions only check a limited number of fields from the json, but they are easily extensible.

One thing to note with this test suite is the inclusion of the global list GLOBAL_NAMES_TO_SKIP. These identifiers are skipped when comparing the global variables of the created and expected jsons. This list of identifiers was added because the globals found when running the plugin locally vs running the plugin through Github Actions did not match. The globals from included standard library files are added to the json by the plugin, and their was a difference between the local and remote standard library files. This difference meant local expected jsons and newly generated jsons on Github Actions would not match, which necessitated the introduction of the list GLOBAL_NAMES_TO_SKIP.

GCC to CAST

Test Python file: automatates/tests/program_analysis/GCC2GrFN/test_gcc_ast_to_cast.py This suite runs the GCC2CAST code on a gcc_ast.json file to produce a CAST object and compares the newly generated CAST with expected CAST. The comparison between new CAST and expected CAST is implemented using the CAST objects __eq__() method.

CAST to AnnCAST

CAST to GrFN