Causal Testing Framework

A Causal Inference-Driven Software Testing Framework

Causal testing is a causal inference-driven framework for functional black-box testing. This framework utilises graphical causal inference (CI) techniques for the specification and functional testing of software from a black-box perspective. In this framework, we use causal directed acyclic graphs (DAGs) to express the anticipated cause-effect relationships amongst the inputs and outputs of the system-under-test and the supporting mathematical framework to design statistical procedures capable of making causal inferences. Each causal test case focuses on the causal effect of an intervention made to the system-under test. That is, a prescribed change to the input configuration of the system-under-test that is expected to cause a change to some output(s).

Installation

Requirements

• Python 3.10, 3.11 and 3.12

• Microsoft Visual C++ 14.0+ (Windows only).

To install the latest stable release of the Causal Testing Framework:

pip install causal-testing-framework

or if you want to install with the development packages/tools:

pip install causal-testing-framework[dev]

Alternatively, you can install directly via source:

git clone https://github.com/CITCOM-project/CausalTestingFramework
cd CausalTestingFramework

then to install a specific release:

git fetch --all --tags --prune
git checkout tags/<tag> -b <branch>
pip install . # For core API only
pip install -e . # For editable install, useful for development work

For more information on how to use the Causal Testing Framework, please refer to our documentation.

Note

We recommend you use a 64 bit OS (standard in most modern machines) as we have had reports of the installation crashing on some 32 bit Debian installations.

Usage

Note

Example usage can be found in the examples directory.

1. To run the causal testing framework, you need some runtime data from your system, some causal test cases, and a causal DAG that specifies the expected causal relationships between the variables in your runtime data (and any other relevant variables that are not recorded in the data but are known to be relevant).

2. If you do not already have causal test cases, you can convert your causal DAG to causal tests by running the following command.

python causal_testing/testing/metamorphic_relation.py --dag_path $PATH_TO_DAG --output_path $PATH_TO_TESTS

3. You can now execute your tests by running the following command.

python -m causal_testing --dag_path $PATH_TO_DAG --data_paths $PATH_TO_DATA --test_config $PATH_TO_TESTS --output $OUTPUT

The results will be saved for inspection in a JSON file located at $OUTPUT. In the future, we hope to add a visualisation tool to assist with this.

How to Cite

If you use our framework in your work, please cite the following:

This research has used version X.Y.Z (software citation) of the Causal Testing Framework (paper citation).

The paper citation should be the Causal Testing Framework paper, and the software citation should contain the specific Figshare DOI of the version used in your work.

BibTeX Citations

Paper

```
@ARTICLE{Clark_etal_2023,
author = {Clark, Andrew G. and Foster, Michael and Prifling, Benedikt and Walkinshaw, Neil and Hierons, Robert M.
and Schmidt, Volker and Turner, Robert D.},
title = {Testing Causality in Scientific Modelling Software},
year = {2023},
publisher = {Association for Computing Machinery},
url = {https://doi.org/10.1145/3607184},
doi = {10.1145/3607184},
journal = {ACM Trans. Softw. Eng. Methodol.},
month = {jul},
keywords = {Software Testing, Causal Testing, Causal Inference}
}
```

Software (example)

```
@ARTICLE{Wild2023,
author = {Foster, Michael and Clark, Andrew G. and Somers, Richard and Wild, Christopher and Allian, Farhad and Hierons, Robert M. and Wagg, David and Walkinshaw, Neil},
title = {CITCOM Software Release},
year = {2023},
month = {nov},
url = {https://orda.shef.ac.uk/articles/software/CITCOM_Software_Release/24427516},
doi = {10.15131/shef.data.24427516.v1}
}
```

How to Contribute

To contribute to our work, please ensure the following:

1. Fork the repository into your own GitHub account, and clone it to your local machine.
2. Create a new branch in your forked repository. Give this branch an appropriate name, and create commits that describe the changes.
3. Push your changes to your new branch in your remote fork, compare with CausalTestingFramework/main, and ensure any conflicts are resolved.
4. Create a draft pull request from your branch, and ensure you have linked it to any relevant issues in your description.

We use the unittest module to develop our tests and the pytest framework as our test discovery, pylint for our code analyser, and black for our code formatting. To find the other (optional) developer dependencies, please check pyproject.toml.

Acknowledgements

The Causal Testing Framework is supported by the UK's Engineering and Physical Sciences Research Council (EPSRC), with the project name CITCOM - "Causal Inference for Testing of Computational Models" under the grant EP/T030526/1.