ScalableCA: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable Software Systems
ScalableCA is a novel and scalable algorithm that demonstrates effectiveness and efficiency in solving large-scale 3-wise CCAG instances. This repository includes the implementation of ScalableCA, the testing instances adopted in the experiments and the experimental results.
The instructions of seting up ScalableCA and validate its general functionality are described in the INSTALL.md file.
The instructions of reproducing the experimental results are described in the EXPERIMENT_REPRODUCTION.md file
ScalableCA is publicly available on Github. To obtain ScalableCA, a user may use git clone to get a local copy of the Github repository:
git clone https://github.com/ScalableCA-Submission/ScalableCA.git
See INSTALL.md.
After building ScalableCA, users may run it with the following command:
./ScalableCA -input_cnf_path [INSTANCE_PATH] -init_2wiseCA_file_path [INITIAL_2wiseCA_PATH] -output_testcase_path [OUTPUT_TESTCASE_PATH] <optional_parameters> <optional_flags>
For the required parameters, we list them as follows.
| Parameter | Allowed Values | Description |
|---|---|---|
-input_cnf_path |
string | path of the input CNF instance |
-init_2wiseCA_file_path |
string | path of the initial 2-wise CA generated by SamplingCA |
-output_testcase_path |
string | path to which the optimized 3-wise CA is saved |
As mentioned above, -input_cnf_path is one of the required parameter and the input file for ScalableCA must be in Dimacs format. The directory named cnf_benchmarks/ contains all 122 testing instances, which can be taken as input. Users may also use FeatureIDE to generate input files from existing configurable systems.
For the optional parameters, we list them as follows.
| Parameter | Allowed Values | Default Value | Description |
|---|---|---|---|
-seed |
integer | 1 | random seed |
-delta |
positive integer | 100 | the number of candidates per round |
-L |
positive integer | 500 | used to control the termination criterion of RALS approach |
-use_fid |
0 or 1 | 1 | whether to use FID technology |
-use_RALS |
0 or 1 | 1 | whether to use RALS approach |
For the optional flags, we list them as follows.
| Flag | Description |
|---|---|
-use_context_aware |
replaces the UGS method with the existing context-aware sampling (CAS) method |
We provide a Python script run.py for users to run ScalableCA in an end-to-end manner. That is, by executing this Python script, users can obtain a 2-wise CA which is initially constructed by SamplingCA and then obtain a 3-wise CA constructed by ScalableCA.
Note: For running SamplingCA separately, users may refer to the instructions in SamplingCA/README.md. For running ScalableCA separately, users may refer to the instructions in the above section.
The usage of run.py is as follows.
python3 run.py [RANDOM_SEED] [INSTANCE_PATH] [OUTPUT_TESTCASE_PATH]
An example of running run.py:
python3 run.py 1 cnf_benchmarks/freebsd-icse11.cnf freebsd-icse11_CA.out
The command above calls SamplingCA to solve the instance cnf_benchmarks/freebsd-icse11.cnf with default hyper-parameter settings, and then calls ScalableCA to generate a 3-wise CA with default hyper-parameter settings. The result is stored in freebsd-icse11_CA.out. And here both SamplingCA and ScalableCA use the random seed of 1.
For non-binary scenarios, effective conversion into binary scenarios can be achieved. This repository includes a converter specifically designed for transforming non-binary scenarios into binary scenarios. The tool supports input in CASA format.
The usage of run.py is as follows.
python3 run.py [RANDOM_SEED] [MODEL_FILE] [CONSTRAINTS_FILE] [OUTPUT_TESTCASE_PATH]
An example of running run.py:
python3 run.py 2 non_binary_instances/Banking1.model non_binary_instances/Banking1.constraints Banking1_CA.out
The directory named src/ includes the implementation of ScalableCA.
The directory named cnf_benchmarks/ contains all 122 large-scale instances and the directory named non_binary_instances/ contains all 25 small-scale non-binary instances. We also provide Benchmark_information.csv which shows the number of options and the number of constraints for large-scale instances. Additionally, non_binary_instances_information.csv is provided, showing the number of options and constraints for small-scale non-binary instances.
The directory named fault_detection_subjects/ contains all 9 subjects used for evaluating fault detection rate.
The file Study_on_t_wise_Coverage.pdf presents both the experimental setup and results on
The directory experimental_results/ contains 6 .csv files for presenting the experimental results.
-
Results_of_Empirical_Study.csv: Empirical Study's Results of the 2-wise CAs generated by SamplingCA and CAmpactor on all instances.
-
Results_of_ScalableCA_and_its_SOTA_competitors.csv: Results of ScalableCA (with SamplingCA as its initialization algorithm) and its state-of-the-art competitors on all testing instances.
-
Results_of_ScalableCA_and_its_alternative_versions.csv: Results of ScalableCA and its alternative versions on all testing instances.
-
Results_of_ScalableCA_with_different_delta_settings.csv: Results of ScalableCA with different
$\delta$ settings on all testing instances. -
Results_of_ScalableCA_with_different_L_settings.csv: Results of ScalableCA with different
$L$ settings on all testing instances. -
T_wise_coverage_of_ScalableCA_and_its_competitors.csv: t-wise coverage (
$4 \leq t \leq 6$ ) of 3-wise CAs by ScalableCA, SamplingCA and CAmpactor, as well as 2-wise CAs by SamplingCA and CAmpactor on all testing instances. -
Results_on_non_binary_small_scale_instances.csv: Results of ScalableCA (with SamplingCA as its initialization algorithm) and its state-of-the-art competitors on 25 non-binary instances.
-
Results_of_fault_detection.csv: Average fault detection rate and average size of 3-wise CAs built by ScalableCA, SamplingCA and CAmpactor, as well as 2-wise CAs constructed by SamplingCA and CAmpactor.
See experiment_reproduction.md.
- Chuan Luo (chuanluophd@outlook.com)
- Shuangyu Lyu (19374290@buaa.edu.cn)
Chuan Luo, Shuangyu Lyu, Qiyuan Zhao, Wei Wu, Hongyu Zhang, Chunming Hu. Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable Systems. Proceedings of ISSTA 2024, 2024.