Which Factor Impacts GUI Traversal-Based Test Case Generation Technique Most? A Controlled Experiment on Android Applications （QRS2017 best paper）
Our experimentation framework is adapted from the ASE 2015 work of Dr.Shauvik Roy Choudhary@ http://bear.cc.gatech.edu/~shauvik/androtest/.
We owe our thanks to their sharing.
1) Setup Process
- Download the box file and install VirtualBox，Vagrant；
- In a terminal, add the Androtest box to Vagrant $ vagrant box add androqrs ./androqrs.box
- Create a directory, say ~/vagrant/androtest，to host the vagrant machine and inside this directory download this Vagrantfile. This file contains the VirtualBox VM configuration that vagrant uses. Note that the configuration defines 10 VM instances labelled run1-run10. $ mkdir -p ~/vagrant/androtest $ cd ~/vagrant/androtest $ wget https://github.com/gongbell/AndroidGUITesting/blob/master/Vagrantfile
- You can start the vm using vagrant up. Vagrant will create all virtual machines on your computer (run1-run10) and start them. To start only one (or few) VMs, pass the VM name(s) as parameters. Once the VM has booted up, login to the VM using SSH. $ vagrant up run1 $ vagrant ssh run1
- Then inside ~/script $ cd script
- Start the script run_allpuma.sh and start running $ bash –x run_allpuma.sh
2) Description of Files and Directories
- In the /home/vagrant/
android-ndk-r10 #--> Android NDK
android-sdk-linux #--> Android SDK
lib #--> Libraries needed by tools
scripts #--> Scripts for experiments (invokes tools)
subjects #--> Open source android app benchmarks
tools #--> Android test input generation tools (include PUMA)
- In the /vagrant(Shared folders)
Vagrantfile #--> Host machine directory (i.e., ~/vagrant/androtest) is mounted as /vagrant on the Vagrant box
projects_34.txt #--> 34 App list for test
run_allpuma.sh #--> Parameter selection，t represents the app runtime, i represents waittime, j represents ui state, k represents search strategy. The meaning of a specific value representation is clearly explained in the script. For example, (0, 0, 0, 0) represents t=0(1hour), i=0(waitforidle), j=0(cosine), k=0(BFS).
3) Files Genearted
test.log – log generated by puma
tool.log -- log generated by the tool
tool.logcat -- logcat from the emulator, while tool was running (contains failure stack traces)
install.log -- installation log of the app on the device
icoverage -- log of intermediate coverage collected
coverage.em -- Emma coverage metadata
coverage.ec or coverage.es -- complete coverage file
coverage[1-11].ec -- snapshots of progressive coverage collected every 5 minutes.
4) Results Extraction
- dataDeal.java –to generate coverage information by extracting line% from coverage.html;
- findCrashByApp.java –to generate the number of failures by extracting Exception info from tool.logcat.
- Usage method:
a. New a project named findCrash(for failure)/ dataDeal(for coverage), then new a class named findCrashByApp(for failure)/ dataDeal(for coverage). Then copy the corresponding .java file in the ~/tools/PUMA/
b. Parameter selection by changing the range of i, j, k; Turn the variable fileName to your local directory.
c. run the project and collect the result.