Capturing benchmark data

[mhdawson]

I've been experimenting on how to easily capture/publish benchmark data. This is the flow I've been thinking would work.

benchmark machine

• (a) -> runs benchmark and publishes to benchmark data machine through simple post using wget

benchmark data machine

• (b) -> accepts post and stores data into database, currently using mysql
• (c) -> once a day generate cumulative graphs
• (d) -> once generated push updated graphs to benchmark.nodejs.org

I have experimental/minimal implementations of a, b and part of c so far.

(b) is implemented with this simple node application here: https://github.com/nodejs/benchmarking/blob/master/experimental/bridge.js which runs on the benchmark data machine. It accepts the connection from the benchmark machine and stores the result in a mysql database. It limits access to the internal ip of the benchmark machine, along with a tls connection and a fixed password. I plan to update this so that it users a certificate instead of the userid/password. At this point it accepts the following url parameters which mirror the database schema (see : for specifics https://github.com/nodejs/benchmarking/blob/master/experimental/dbmachine_setup.txt)

• streamid - release stream (1 for master, 2, for 5.x, etc.)
• benchid - id for the benchmark (1 for footprint, etc)
• time - timestamp for the benchmark run
• value - score for the benchmark

This shell script is used to post the result https://raw.githubusercontent.com/nodejs/benchmarking/master/experimental/postit.sh

This job (https://ci.nodejs.org/job/benchmark-footprint-experimental/) is an experimental version of (a) which runs the existing footprint micro-benchmark .../benchmark/report-startup-memory.js and publishes the result to the benchmark data machine.

(c) is partially implemented in that the files in https://github.com/nodejs/benchmarking/tree/master/experimental/chartGen can be used with phantomjs to generate the graph from the database. With the following being the current output:

Overall this approach would let us publish benchmark results with a small amount of code, results would be persistent as they are published as a png and it will be relatively easy to add new graphs.

Once we get agreement that this approach seems reasonable next steps include:

1. clean up the sampale code/scripts, check into the benchmarking repo in a non-experimental directory
2. get agreement from the build group to publish to benchmarking.nodejs.org along with the required public keys to be able to do the pushes
3. create base index file for benchmarking.nodejs.org that provides intro/links/in-lines images of initial benchmarks
4. create cron job to generate push updated graphs daily
5. create/update ansible scripts (used by build group to configure machines) to install additional dependencies and components in a repeatable way on the benchmark machines
6. Figure out backup strategy for benchmark database
7. Expand to additional benchmarks

[mhdawson]

Opened this issue on the build workgroup repo to get the discussion started about pushing to nodejs.org
nodejs/build#281

[mhdawson]

@nodejs/benchmarking please review/comment

[mhdawson]

Was discussed in last Benchmarking WG meeting with no objects to approach. Will push forward. Have moved most of this info to https://github.com/nodejs/benchmarking/blob/master/benchmarks/README.md so closing