Feature request: benchmark runner progress indicator

[mscdex]

• Version: n/a
• Platform: n/a
• Subsystem: benchmark

Since the benchmark runner defaults to 30 runs and some benchmarks can take awhile to run, it would be nice to have a progress indicator on stderr when the CSV data is being written to non-TTY (e.g. piped to R) on stdout (similar to how cURL displays download progress).

Also, changing the benchmark running logic to instead alternate the benchmark parameter combinations between the new and old executables will allow for faster displaying of results. This may require changes to the R script if it buffers its results until data for all benchmark parameter combinations are received.

[AndreasMadsen]

Also, changing the benchmark running logic to instead alternate the benchmark parameter combinations between the new and old executables will allow for faster displaying of results.

Not sure I understand, do you want to change this:

const queue = [];
for (let iter = 0; iter < runs; iter++) {
  for (const filename of benchmarks) {
    for (const binary of binaries) {
      queue.push({ binary, filename, iter });
    }
  }
}

to this:

const queue = [];
for (const filename of benchmarks) {
  for (let iter = 0; iter < runs; iter++) {
    for (const binary of binaries) {
      queue.push({ binary, filename, iter });
    }
  }
}

? That should be fine, I was thinking about that too.

his may require changes to the R script if it buffers its results until data for all benchmark parameter combinations are received.

The R script definitely buffers the result. I have no idea how to make it a stream processor. Also I strongly recommend always saving the results, so at the very least you should run:

node ./benchmark/compare.js --new ./node-pr --old ./node-master -- buffer \
  | tee results.csv \
  | Rscript ./benchmark/compare.R

[mscdex]

Regarding the benchmark execution order, yes, either the second code block or a variation of the second block where the two inner for-loops are swapped, doesn't matter really.