This repository is for the research on exploratory testing of AI applications conducted by the team led by Professor Hong Zhu at Oxford Brookes University, Oxford, UK. It is a part of the research project on datamorphic software testing methodology for testing AI applications.
- Table of Contents
Morphy -- The folder contains the automated software testing tool Morphy
-- Morphy.bat batch script file to start Morphy on Windows OS
-- Morphy.jar executable jar file
-- MorphyAnnotations.jar package jar file for building morphy test code
-- icons the folder contains the icons used by the Morphy GUI
Experiments -- The folder contains files used for conducting the control experiments with the exploaratory testing of classifiers
-- src The Java source code of the classifiers and the test code
-- bin The class files of the test code of the classifiers and the test code
-- TestScripts The test scripts for executing the experiments
-- Experiment data The data generated by executing the test scripts. They are the result data of the experiments
CaseStudy -- The folder contains all files of the case studies with the exploratory testing of machine learning applications
-- BankChurners The folder contains all files of the case study on the bank churners prediction application
-- Mushroom The folder contains all files of the case study on classification of mushrooms on their edibility
-- RedWineQuality The folder contains all files of the case study on classification of red wine according to their quality
For each of these folders, it contains the following subdirectories: (where XXX is Bank, RedWine or Mushroom)
-- Builders The original dataset and Python code to construct the models using various machine learning algorithms.
---- XXX_Build_Models.py The Python code to build various ML models
---- XXX.csv The original dataset in csv format used as the input to build models
---- XXXV2.csv The original dataset with the headline removed for used in morphy test code as input
---- XXXTrainingData The dataset in the Morphy's internal representation. It contains only the input data, i.e. the labelled output is excluded.
-- models The folder contains the models produced by the Python code
---- Model Info.txt The output of the Python code about the accuracy of the modles
---- XXX Model Info.xdoc The information of the models' accuracy in MS Word format, etc.
---- YYY_model.sav The model generated by the Python code, where YYY is the name of the model.
-- PyScript The folder contains Python script that are invoked by the test scripts to execute the predictions on input data. Each invokes on model.
-- src The source code of the test system in Morphy
-- TestScript The test scripts for conduct various experiments in the case study using Morpthy.
Execution Time Data -- The folder contains the data collected in the experiments on execution times of the test strategies. The test scripts for the experiements are also given in this folder, but in the sub-folder of Test Scripts.
-- Coded Classifiers : Data of the execution time lengths and test scripts for testing coded classifiers.
-- Red Wine Quality: Data of the execution time lengths and the test scripts for testing the machine learning models of red wine quality.
-- Muchroom: Data of the execution time lengths and the test scripts for testing the machine learning models of mushroom editbility.
-- Bank Churners: Data of the execution time lengths and the test scripts for testing the machine learning models of bank churner prediction.
- Instructions
2.1. Installation and start of Morphy
(1) Download the files in the Morphy folder
(2) Copy the files to the directory "C:\Morphy"
(3) Double clisk on the Morphy.bat file, the Morphy GUI will show on screen.
2.2. Repeat the Controlled Experiments
(1) Setup for the Control Experiments by copy files in the Experiment\bin to the "C\Morphy" folder
(2) Start the Morphy tool and execute the test scripts in the folders Experiments\TestScripts(Directed Walk | Random Walk | Random Target)
(3) The experiment data will be output to the csv file "ExpDataFile.csv". Rename the files to your preferred file name.
2.3. Repeat the Case Studies
(1) Download the files in the folder CaseStudy\BankChurners (Mushroom or RedWineQuality) and copy all files to the C:\Morphy folder.
(2) Start the Morphy tool and load test spec file BankChurners_TestSpec.class to Morphy tool.
(3) Execute the test script Bank-RandomTarget-Allmodels.scr for random target strategy. Similarly, use other test scripts for different test strategies.
(4) The data will be saved in the file Bank-ExpDataFile.csv
2.4. Recompile the test system
(1) Use a Java IDE, such as Eclipse to create a Java project
(2) Copy the Java source code .java files in the CaseStudy\BankChurners\src folder to the IDE
(3) Add the Morphy.jar file in the Morphy folder to the libray of the Java build path.
(4) Compile the Java source code to generate .class files
- Analysis of Data
3.1. Analysing Data of Controlled Experiments:
Copy the data that you obtained from running the test scripts to the Data tab of the corresponding ExpData-StrategyName-Analysis.xlsx file. The analysis results will be automaticalled shown in the other tabs.
3.2. Analysing Data of Case Studies:
Copy the data from the ExpDataFile that generated by running the test script to the Original Data tab of the Classifier-ExpDataFile-Analysis-Template.xlsx file. The results of the analysis will be automatically generated in the tabs of Runs, Mutants, Effectiveness and Capability.