Program Analysis Testing

Testing Suites

Here we outline the general structure of the newly created (Spring 2022) testing suites. All of these testing suite use pytest's parameterize functionality to automatically create tests based on test data. The structure of each test is roughly as follows: Each test file defines a constant TEST_DATA_DIR where all test data is stored.

• Each test uses two files for test data
  1. An input file from the previous step of the PA pipeline
  2. the expected result file for the step of the pipeline being tested
• Tests for the above pairs of files are dynamically generated using pytest's parameterize functionality based on a test names. To make this work correctly, the expectation is that the input file and expected result file will follow an expected pattern. In each test file we specify the exact pattern, but in general it is something like name_of_test_{input_file_suffix} and name_of_test_{expected_file_suffix}. To add new tests to a suite of tests, one must simply add the input file and expected result to TEST_DATA_DIR and the testing suite will automatically collect it as a new test case.

GCC Plugin

Test Python file: automates/tests/program_analysis/GCC2GrFN/test_gcc_plugin_c.py This suite runs the GCC plugin to create a gcc_ast.json file, and compares the created file from an expected json file. To perform the comparison, we the SimpleNamespace from Python's types module. Basically, SimpleNamespace allows us to use dot . access on elements of the json instead of dictionary like access. In the test file, we define functions to compare the generated gcc json with the expected gcc json. These comparison functions only check a limited number of fields from the json, but they are easily extensible.

One thing to note with this test suite is the inclusion of the global list GLOBAL_NAMES_TO_SKIP. These identifiers are skipped when comparing the global variables of the created and expected jsons. This list of identifiers was added because the globals found when running the plugin locally vs running the plugin through Github Actions did not match. The globals from included standard library files are added to the json by the plugin, and their was a difference between the local and remote standard library files. This difference meant local expected jsons and newly generated jsons on Github Actions would not match, which necessitated the introduction of the list GLOBAL_NAMES_TO_SKIP.

GCC to CAST

Test Python file: automates/tests/program_analysis/GCC2GrFN/test_gcc_ast_to_cast.py This suite runs the GCC2CAST code on a gcc_ast.json file to produce a CAST object and compares the newly generated CAST with expected CAST. The comparison between new CAST and expected CAST is implemented using the CAST objects __eq__() method.

CAST to AnnCAST

Test Python file: automates/tests/program_analysis/CAST2GrFN/test_cast_to_ann_cast.py This suite runs the AnnCAST pipeline on an input CAST json and compares it with an expected PipelineState object. The expected PipelineState object is stored as a binary "pickled" file. The pickling is performed using the dill library, since the standard pickle library does not allow pickling of lambdas.
The comparison of the new PipelineState object and the expected PipelineState object is performed using the PipelinState's equiv() method. The method should be extended in the future to check more things.

CAST to AnnCAST GrFN 2.2

Test Python file: automates/tests/program_analysis/CAST2GrFN/test_cast_to_ann_cast_grfn_2_2.py This suite is the same as CAST to AnnCAST, but runs the AnnCAST pipeline using the GENERATE_GRFN_2_2 flag set to True.

CAST to GrFN via AnnCAST pipeline

Test Python file: automates/tests/program_analysis/CAST2GrFN/test_cast_to_grfn_ann_cast_pipeline.py This suite runs the AnnCAST pipeline on an input CAST json and compares the resulting GrFN stored in the final PipelineState against an expected GrFN. The comparison between new GrFN object and expected GrFN object is performed using the __eq__() method of the GroundedFunctionNetwork class. Once execution is fully implemented using the new AnnCAST pipeline, this __eq__() method should be extended to compare the execution results of the new and expected GrFN.

CAST to GrFN 2.2 via AnnCAST pipeline

Test Python file: automates/tests/program_analysis/CAST2GrFN/test_cast_to_grfn_ann_cast_pipeline_grfn_2_2.py This suite is the same as CAST to GrFN via AnnCAST pipeline, but runs the AnnCAST pipeline using the GENERATE_GRFN_2_2 flag set to True.