This repository belongs to our submitted manuscript:
CC: Causality-Aware Coverage Criterion for Deep Neural Networks
You can find the code for our paper in this repository. Also, you can find the archived version of this repository on Software Heritage
Deep neural network (DNN) testing approaches have grown fast in recent years to test the correctness and robustness of DNNs. In particular, DNN coverage criteria are frequently used to evaluate the quality of a test suite, and a number of coverage criteria based on neuron-wise, layer-wise, and path-trace-wise coverage patterns have been published to date. However, we see that existing criteria are insufficient to represent how one neuron would influence subsequent neurons; hence, we lack a concept of how neurons, when functioning as causes and effects, might jointly make a DNN prediction.
On the basis of recent advances in interpreting DNN internals using causal inference, we present the first causality-aware DNN coverage criterion, which evaluates a test suite by quantifying the extent to which the suite provides new causal relations for testing DNNs. Performing standard causal inference on DNNs presents both theoretical and practical hurdles. We introduce CC (causal coverage), a practical and efficient coverage criterion that integrates a number of optimizations using DNN domain-specific knowledge. We illustrate the efficacy of CC utilizing both diverse, real-world test inputs and adversarial inputs, such as adversarial examples (AEs) and backdoor inputs. We demonstrate that CC outperforms previous DNN criteria under various settings with moderate cost.
python 3.9
torch
torchvision
pyflann
scipy
sklearn
numpy
causallearn
torchattacks
We also provide a ref_env.yml file for you to automatically install the dependencies.
# default run CC on CIFAR10
python run.py
# use following command to see more args
python run.py --help