📝 "Synthesizing Benchmarks for Predictive Modeling" (CGO Best Paper 2017)
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
BUILD Add a BUILD file. Jul 17, 2018
README.md Add GPLv3 license. Jul 11, 2018


Synthesizing Benchmarks for Predictive Modeling

Chris Cummins, Pavlos Petoumenos, Zheng Wang, Hugh Leather.

Winner of Best Paper Award CGO'17


Predictive modeling using machine learning is an effective method for building compiler heuristics, but there is a shortage of benchmarks. Typical machine learning experiments outside of the compilation field train over thousands or millions of examples. In machine learning for compilers, however, there are typically only a few dozen common benchmarks available. This limits the quality of learned models, as they have very sparse training data for what are often high-dimensional feature spaces. What is needed is a way to generate an unbounded number of training programs that finely cover the feature space. At the same time the generated programs must be similar to the types of programs that human developers actually write, otherwise the learning will target the wrong parts of the feature space.

We mine open source repositories for program fragments and apply deep learning techniques to automatically construct models for how humans write programs. We then sample the models to generate an unbounded number of runnable training programs, covering the feature space ever more finely. The quality of the programs is such that even human developers struggle to distinguish our generated programs from hand-written code.

We use our generator for OpenCL programs, CLgen, to automatically synthesize thousands of programs and show that learning over these improves the performance of a state of the art predictive model by 1.27x. In addition, the fine covering of the feature space automatically exposes weaknesses in the feature design which are invisible with the sparse training examples from existing benchmark suites. Correcting these weaknesses further increases performance by 4.30x.

Keywords Synthetic program generation, OpenCL, Benchmarking, Deep Learning, GPUs

  title={Synthesizing Benchmarks for Predictive Modeling},
  author={Cummins, Chris and Petoumenos, Pavlos and Wang, Zheng and Leather, Hugh},


The code for this paper (everything in the directory code) is released under the terms of the GPLv3 license. See LICENSE for details. Everything else (i.e. the LaTeX sources and data sets) are unlicensed, please contact Chris Cummins chrisc.101@gmail.com before using.