GSoC Draft

tobami edited this page Mar 11, 2011 · 5 revisions

There are a number of subprojects that need to be completed for Codespeed to become a standard tool in the continuous integration toolbox arsenal:

Integration

  • Completing VC integration (Github, Bitbucket) [Maybe not necessary, as it is nearly complete by now...]
  • Buildbot plugin that uses idle time on buildslaves, and only if it is available, to run benchmarks until a desired statistical confidence is reached. (continuous integration)

Visualization and Evaluation Support

  • Think about a way to cleanly save and display branches. They may only need to be compared to trunk once or twice, and then discarded.
  • Better statistics handling: error bars, box or violin plots. Color too high stddev?
  • Giving the user some way to define 'views'/diagrams that are interesting for a particular purpose. Perhaps a set of standard presets for diagram types and a way to define some standard manipulations of the data, like normalization, sorting and grouping.
  • Porting to new jqPlot version or to other js graphics lib (Necessary to eliminate some bugs and to display error bars and other more statistical plot types)
  • More series aggregation
  • PDF/Image export of custom diagrams
  • proper handling of failed benchmark runs
    • visual indication of a complete failure
    • visual indication when it has sporadical failures

Usability/Setup

  • Supporting multi project instance well. For example to be able to use a single Codespeed instance to host numbers for all Django apps, or all Python implementations, or all JavaScript VM implementations.

General Improvements

  • Better caching/performance for large datasets