Skip to content
Code to reproduce simulations and plots for "Sequential estimation of quantiles with applications to A/B-testing and best-arm identification"
C++ Python CMake R Makefile M4 Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

This repository contains an R pacakge with all code necessary to reproduce simulation results and all plots from

Howard, S. R. and Ramdas, A. (2019), Sequential estimation of quantiles with applications to A/B-testing and best-arm identification, preprint, arXiv:1906.09712.

You can install the package like so:


Building the simulation code requires a C++ compiler with C++14 support.

All output files are written to the build/ subdirectory of the working directory. Simulations take a while to run; my results CSVs are under inst/extdata. You can grab the installed paths to these files with

# For Figure 5:
system.file('extdata/simulations_10_64.csv', package='quantilecs')
# For Figure 6:
system.file('extdata/simulations_2_256.csv', package='quantilecs')
  • make_intro_plot(save=TRUE) reproduces Figure 1.
  • plot_quantile_cs(save=TRUE) reproduces Figure 2.
  • plot_tuning(save=TRUE) reproduces Figure 3.
  • run_paper_simulations(save=TRUE) runs the simulations for Figure 5, writing results to build/simulations_10_64.csv. This takes a couple of hours on my four-core MacBook Pro.
  • run_ab_simulations(save=TRUE) runs the simulations for Figure 6, writing results to build/simulations_2_256.csv. This also takes some time.
  • make_simulation_plots('path/to/csv', save=TRUE) reproduces Figure 5.
  • make_ab_plots('path/to/csv', save=TRUE) reproduces Figure 6.

To run the C++ unit tests, run make -C test runtests at the command line. You will need to have the BH and confseq R packages installed, since we link against C++ headers from those libraries.

You can’t perform that action at this time.