Filip Krikava & Jan Vitek
This is the artifact for the paper Tests from Traces: Automated Unit Test Generation for R published at ISSTA 2018. The aim is to:
- show how the genthat tool introduced in the paper works and can be used for extracting code from R packages, and
- reproduce the experiment used to evaluate it.
Concretely, we rerun the all the code snippets shown in Section 3 of the paper and the evaluation from Section 4.
The artifact is composed of two parts:
- Genthat Overview
- CRAN Experiment
The first part corresponds to the Section 3 of the paper where we introduce the tool. The second part reproduces the experiment that we used to evaluate the tool in Section 4 and reproduce Table 1, 2 and Figure 4.
- about 8GB of disk space
Running the artifact
- Clone the repository fikovnik/ISSTA18-artifact:
git clone https://github.com/fikovnik/ISSTA18-artifact
The artifact is provided as a docker image
The image contains R 3.4 with a web version of Rstudio and a set of preinstalled R packages.
- Run the docker image:
from the cloned repository.
This runs the docker container in foreground making it easy to close by just issuing CTRL+C or simply closing the terminal window.
It will download the already prebuilt docker image from docker hub.
If you prefer to build your own, please follow the instructions at the end of this document.
If you ran the docker image manually, please make sure its name is
The rest of the commands shall be done in new terminal window.
Please note that both evaluation parts bellow require the docker image to be running.
This part essentially explains how genthat works and reproduces code examples that are presented in the paper. There are two ways to run it:
In an Rstudio that is available on http://localhost:8787 after the docker container is started. The username and password is
rstudio. In the Rstudio application, please open the
overview.Rmdfile. A Rmarkdown or Rmd is simply a text file in markdown-like format with snippets of R code that can be run (knitted - terminology used in Rmarkdown) to produce an output with the results of the code evaluation embedded within the document. Running a Rmd file can be done either interactively by running the snippets chunk-by-chunk (using the green play button next to each snippet) or by knitting the whole document by pressing Ctrl+Alt+K (Command+Shift+K on OSX).
In a terminal by running:
docker exec -u rstudio -w /home/rstudio issta18-genthat Rscript -e 'rmarkdown::render("overview.Rmd")'
from the cloned repository.
In both cases the results are in
overview.html will be overwritten.
CRAN experiment (Section 4)
The Section 4 presents an experiment that runs genthat on 1547 CRAN packages. In this artifact, we present the same experiment, with the difference that only 857 packages are selected. The reason is that tracing, as indicated in the paper, takes a long time to run. Running the 1547 took 1d 18h on two virtual nodes with 60GB RAM and 16 CPU at 2.2GHz. On an average laptop or desktop, it might take much more. The packages selected for this artifact represent all the original ones for which tracing (on the two nodes) took less than 2 minutes.
This experiment is composed in a way that gives some flexibility as how much time one wants to devote to the reproduction. We have created the following package files (package name per line) with different sizes:
packages-100.txt~ 100 packages, completes in ~30min
packages-200.txt~ 200 packages, completes in ~1h 15min
packages-400.txt~ 400 packages, completes in ~2h 30min
packages-all.txt~ all packages, completes in ~4h 15min
which simply restrict the selection of packages.
The time needed was measured on i7 laptop with 16GB of RAM.
The experiment can also be run iteratively, each time adding more packages.
It does not rerun packages that completed.
This is why it is necessary to remove the
experiment directory before an attempt to redo the experiment (cf. bellow).
By default the experiment is set to run all packages.
This can be changed this by symlinking the
packages.txt to a different target set.
It is possible to try genthat on any package.
Simply just add your favorite R package(s) into
Just make sure that they are installed including examples and tests:
install.packages("MyFavoritePackage", destdir='/CRAN', INSTALL_opts=c('--example', '--install-tests', '--with-keep.source', '--no-multiarch'), dependencies=TRUE)
and the downloaded sources in
/CRAN/*.tar.gz are extracted in
Also, please make sure that you read the note on running package code bellow.
Rerunning the experiment
The repository already contains a completed experiment (in
To redo the experiment:
Make sure the docker image is running
Remove the existing results
rm -fr experiment
- Select the size of the experiment, for example, to run a smaller set do:
ln -sf packages-400.txt packages.txt
- Start the experiment by running:
This runs the
./extract-tests.R it in parallel for each package found in the
The result are two RDS files:
experiment/<package_name>/coverage.RDS- contains test code coverage from covr
experiment/<package_name>/tests.RDS- contains the result of running
plus all the regular genthat extracted tests.
This will take a while!
The progress can be also followed by tailing the parallel log output in
./tail -f experiment.log Seq Host Starttime JobRuntime Send Receive Exitval Signal Command 4 : 1526026700.641 2.260 0 0 0 0 ./extract-tests.R R.matlab 1 : 1526026700.621 130.101 0 0 0 0 ./extract-tests.R elliplot ... ... ... 11 : 1526026929.474 2695.841 0 0 -1 15 ./extract-tests.R sparklyr 12 : 1526027007.903 238.169 0 0 0 0 ./extract-tests.R mlearning ... ... ...
By default each package is run with 10 minutes time out (the
--timeout argument to parallel in
If you see that many packages time outed (
Exitval is -1 and
experiment.log, like the one with
Seq 11 in the above listing), please increase the time out in the script.
- After the experiment has ran, knit the
cran-experiment.Rmdeither in R studio or by running:
docker exec -u rstudio -w /home/rstudio issta18-genthat Rscript -e 'rmarkdown::render("cran-experiment.Rmd")'
and then open the resulting
A note on running package code
Genthat extracts tests by running available R code from R packages. This is not always deterministic since some of the code tries to access various web pages or have some hard-coded values. This is means that it might happen that some of the packages might fail. However, there should be a small number of them. This also means that the results might be slightly different depending on which code managed to run successfully.
Build image locally
The image is hosted on the docker hub and shall be available for download.
To make a local build run
#!/bin/sh docker build \ --rm \ --build-arg REPO=https://mirrors.nic.cz/R \ -t prlprg/issta18-genthat \ .
This will take about 2h to run depending on the network speed.
Please adjust the
REPO to be a one of your local mirror.
Docker on OSX
It is better to use homebrew cask to install docker:
brew cask install docker
in case you see
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. error message
Docker on Linux
In some distribution the package does not add the current user to
In this case, either add yourself to
docker group or run the
./run* commands with