Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
176 lines (135 sloc) 5.77 KB

Benchmarking Notes

This contains notes on running benchmarks on these binaries. Generally this is test the speed of QEMU and it’s emulation code.

Binaries To Test

Here we have several tables of binaries to test. We refer to the current benchmarking set from the next stage, Run Benchmark.

For a final test we might compare the system QEMU with a reference build as well as our current build.

Binarytitle
/usr/bin/qemu-aarch64system-2.5.log
~/lsrc/qemu/qemu-builddirs/arm-targets.build/aarch64-linux-user/qemu-aarch64master.log
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64softfloat-v4.log

We might want to copy QEMU across at the start of each patch we re-factor so we can cleanly test without getting lost in the changes of a large series.

Binarytitle
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64.startbaseline.log
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64development.log
~/lsrc/qemu/qemu-builddirs/arm-targets.build/aarch64-linux-user/qemu-aarch64master.log

In fact while developing we can just keep running against the current build and comparing against previously recorded logs.

Binarytitle
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64development.log

Run Benchmark

This runs the benchmark against each binary we have selected above.

import subprocess
import os

runs=[]

for qemu,logname in files:
   cmd="taskset -c 0 %s ./vector-benchmark -n %s | tee %s" % (qemu, tests, logname)
   subprocess.call(cmd, shell=True)
   runs.append(logname)

return runs

Graph Results

Before we graph the results we need to import the data from the run into a table of data we can now feed into gnuplot.

For later processing we need a simple count of the number of tests we actually ran. We could calculate this from the variable tests in Run Benchmark but it is just as simple to count the lines of output.

cat $data | wc -l

This fragment munges the run data (which we did asynchronously) into the org-mode table from which we can then use to feed gnuplot.

nruns = len(data)

table = [[0 for x in range(nruns+1)] for y in range(ntests+1)]
table[0][0] = "test/run"

for logfile, col in zip(data, xrange(1, nruns+1)):
     table[0][col] = "%s" % (logfile.rstrip(".log"))
     with open(logfile) as f:
          row = xrange(1, ntests+1)
          for row, line in enumerate(f, 1):
             (test,result) = line.split(",")
             table[row][0] = test
             table[row][col] = result.rstrip()

return table

Finally with the results nicely tabulated we can call GNUPlot with the data to plot our nice bar charts.

set title "Vector Benchmark Results (lower is better)"
set style data histograms
set style fill solid 1.0 border lt -1

set xtics rotate by 90 right
set yrange [:]
set xlabel noenhanced
set ylabel "nsecs/Kop" noenhanced
set xtics noenhanced
set ytics noenhanced
set boxwidth 1
set xtics format ""
set xtics scale 0
set grid ytics
set term pngcairo size 1200,500

plot for [i=2:5] data using i:xtic(1) title columnhead

Upload PNG

This is a simple restful script to upload to imgur. While restful-mode is intended for live checking of REST APIs it can easily be embedded in an org-babel source block.

:client_id = fd2da649643c743

# Upload images to imgur
POST https://api.imgur.com/3/image
Authorization: Client-ID :client_id
Content-type: image/png

< benchmark.png

This is a simple helper function to call the above code but strip out the URL so you don’t need to hunt through all the headers.

(when (string-match
       (rx "link" (one-or-more (any "\":" whitespace))
           (group (one-or-more (not (any "\"")))))
       json-string)
  (match-string 1 json-string))