An implement for paper "QuanForge: A Mutation Testing Framework for Quantum Neural Networks"
The repository is structured as follows:
|-- adversarial/
|-- models/
|-- mutation/
|-- tools/
|-- adv_baseline.py
|-- config.py
|-- mutation_testing.py
|-- requirements.txt
|-- train.py
Related packages are listed in requirements.txt. We used Python 3.10 to construct a conda environment.
pip install -r requirements.txt
Since model training and mutant generation need time, here we provide some QNNs and mutants used in our experiments for a quick start.
They are put in directory ~\data\models and ~\data\mutation respectively.
For example, to conduct RQ1, you can run:
python mutation_testing.py --dataset mnist --structure qcl --resize --finite 1000
The mutation score of the original test suite will be output.
python train.py --dataset mnist --structure qcl --class_idx 0 1 --resize --data_scale 0.2 --finite 1000
Options:
--dataset: Target dataset, choices: mnist (default), fashion_mnist.
--structure: Quantum neural networks, choices: qcl (default), qcnn, hcqc, drnn. For HCQC, ansatz can choosed from U_TTN, U_15, U_SO4 (default), U_6 and U_SU4 by setting --ansatz.
--class_idx: Target classes of dataset used for binary or ternary classification. Ternary classification is recommended to implement on mnist. default: 0 1.
--resize: Rescale the image size to a smaller size to fit the circuit scale. Here is (8, 8) for 6-qubit DRNN and (16, 16) for other 8-qubit QNNs.
--data_scale: Percentage of whole dataset used for model training. Recommended as 0.2.
--finite: The number of shots for measurement. default: 1000. 0 means an ideal simulator with infinite shots.
--noise: Optional config. Add noise channels defined in models/noise_channels.py to circuits by using default.mixed device of PennyLane.
Mutation testing for original test suite using default settings.
python mutation_testing.py --dataset mnist --structure qcl --resize --finite 1000
You can change settings of target gates and scope in mutation/config_example.py, operator type in mutation_testing.py.
By default, use Random Gate Addition, qubit range: list(range(n_qubits)), depth range: [0~1]
python mutation_testing.py --dataset mnist --structure qcl --resize --finite 1000
To load different test suites, you can add optional config:
--hconf: high-confidence inputs (far from decision boundary).
--bias_pro: skewed, class-imbalanced inputs. 10 1 used in the paper for binary task and 10 1 1 for ternary task.
--num_test: small, smaller-scale suite. 50 used in the paper.
--lconf: low-confidence inputs (close to decision boundary).
--ood: out-of-distribution inputs.
--aug: augmented inputs. Augment techniques are defined in tools/custom_data_transform.py.
To obtain adversarial examples generated by FGSM, you need to first run:
python adv_baseline.py --dataset mnist --structure qcl --resize --finite 1000 --attack FGSM
Generated figures will be saved in /data/analysis/adversarial/{dataset}/{structure}. After this, you can append --with_adv to obtain the score of Adv suite.
Change the depth_range in mutation/config_example.py and re-run the basic command.
Change the qubit_range in mutation/config_example.py and re-run the basic command.
Change the gate_types in mutation/config_example.py and re-run the basic command.
Change the configurations of sem_t and trivial_t (Lines 176 and 177) and re-run the basic command.
Noise channels are defined in models/noise_channels.py. We have provided the noisy version of HCQC (ansatz U_TTN). You can also try other QNNs by adding --noise when training.
python mutation_testing.py --dataset mnist --structure hcqc --ansatz U_TTN --resize --finite 1000 --noise --num_test 50
Note that here --num_test is set as 50 due to the computational limitations of device.