Pilot Benchmark Framework: Get good results fast
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
build-env
cli
cmake
doc
examples
include
lib
packaging
.gitignore
AUTHORS
CMakeLists.txt
CTestCustom.cmake
LICENSE
README.md
lgpl-2.1.txt
pilot-tool.doxyfile

README.md

INTRODUCTION

The Pilot Benchmark Framework provides a tool (bench) and a library (libpilot) to automate computer performance measurement tasks. This is a project under active development. The project is designed to answer questions like these that rise often from performance measurement:

  • How long shall I run this benchmark to get a precise result?
  • Is this new scheduler really 3% faster than our old one or is that a measurement error?
  • Which setting is faster for my database, 20 worker threads or 25 threads?

Our design goals include:

  • Be as intelligent as possible so users do not have to go through rigorous statistics or computer science training.
  • Results must be statistical valid (accurate, precise, and repeatable).
  • Using the shortest possible time.

To achieve these goals Pilot has the following functions:

  • Automatically deciding what statistical analysis method is suitable based on system configuration, measuring requirements, and existing results.
  • Deciding the shortest length for running a benchmark to achieve the desired measurement goal such as confidence interval.
  • Save the test results in a standard format for sharing and comparing results.
  • And much more...

To learn how to use Pilot or if you want to refresh the statistics, please read the documentation: https://docs.ascar.io/

Pilot is written in C++ for doing fast in place analysis. It is released in dual BSD 3-clause and GPLv2+ license.

Our recommended channel for support is Slack: https://join.slack.com/t/tuneupai/shared_invite/enQtMzE4MDAzNTUzODQwLTRhMjM2MmQ0MzViYjc2NThkZTNkZTQzZjhmZWUzYzI1MjU2MzNjMTE5MmJjZjY0MjU5YTk4NDVhMTc0NWJhMmY. You can also use the following mailing lists:

INSTALL

Installation guide: https://docs.ascar.io/install.html

RUN IT

Please see the tutorials: https://docs.ascar.io/tutorial-list.html

DEVELOPMENT

We partially follow Google's C++ coding standard: https://google-styleguide.googlecode.com/svn/trunk/cppguide.html . The exception is we use four spaces for indentation (Google's style uses two spaces).

ACKNOWLEDGMENTS

This is a research project from the Storage Systems Research Center in UC Santa Cruz. This research was supported in part by the National Science Foundation under awards IIP-1266400, CCF-1219163, CNS-1018928, CNS-1528179, by the Department of Energy under award DE-FC02-10ER26017/DESC0005417, by a Symantec Graduate Fellowship, by a grant from Intel Corporation, and by industrial members of the Center for Research in Storage Systems. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.