Skip to content
Code to reproduce simulations and plots for "Sequential estimation of quantiles with applications to A/B-testing and best-arm identification"
C++ Python CMake R Makefile M4 Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R
inst/extdata
src
test
.gitignore
DESCRIPTION
LICENSE
NAMESPACE
README.md

README.md

This repository contains an R pacakge with all code necessary to reproduce simulation results and all plots from

Howard, S. R. and Ramdas, A. (2019), Sequential estimation of quantiles with applications to A/B-testing and best-arm identification, preprint, arXiv:1906.09712.

You can install the package like so:

install.packages('devtools')
devtools::install_github('gostevehoward/quantilecs')

Building the simulation code requires a C++ compiler with C++14 support.

All output files are written to the build/ subdirectory of the working directory. Simulations take a while to run; my results CSVs are under inst/extdata. You can grab the installed paths to these files with

# For Figure 5:
system.file('extdata/simulations_10_64.csv', package='quantilecs')
# For Figure 6:
system.file('extdata/simulations_2_256.csv', package='quantilecs')
  • make_intro_plot(save=TRUE) reproduces Figure 1.
  • plot_quantile_cs(save=TRUE) reproduces Figure 2.
  • plot_tuning(save=TRUE) reproduces Figure 3.
  • run_paper_simulations(save=TRUE) runs the simulations for Figure 5, writing results to build/simulations_10_64.csv. This takes a couple of hours on my four-core MacBook Pro.
  • run_ab_simulations(save=TRUE) runs the simulations for Figure 6, writing results to build/simulations_2_256.csv. This also takes some time.
  • make_simulation_plots('path/to/csv', save=TRUE) reproduces Figure 5.
  • make_ab_plots('path/to/csv', save=TRUE) reproduces Figure 6.

To run the C++ unit tests, run make -C test runtests at the command line. You will need to have the BH and confseq R packages installed, since we link against C++ headers from those libraries.

You can’t perform that action at this time.