Add benchmark runner and plot scripts, plus big readme

.gitignore

@@ -1,3 +1,7 @@
 /bench
 /bench.dat
+/machines/*
 /README.html
+/results.csv
+/results.svg
+/Rplots.pdf

README.md

@@ -1,4 +1,110 @@
-bench.c
-========
-Microbenchmark that executes repeated writes and fsyncs of the requested size.
+This is a set of benchmarking utilities for measuring the latencies of disks
+(mostly interesting for SSDs).
 
+The lowest level tool is the `bench` program. This is a microbenchmark that
+executes repeated writes and fsyncs of the requested size. To build it and see
+its usage, run:
+
+    make bench
+    ./bench --help
+
+A possibly outdated version of that message is:
+
+    Time how long it takes to append into a file and fdatasync it.
+    The results are printed in seconds total for all the writes.
+
+    Usage: ./bench [options]
+      --count=NUM      Number of sequential appends to measure [default 1000]
+      --direct=yes|no  Whether to use O_DIRECT [default no]
+      --file=NAME      File to create/truncate and write into [default bench.dat]
+      --help           Print this help message and exit
+      --offset=BYTES   Number of bytes to skip at start of file [default 0]
+      --size=BYTES     Number of bytes to append in each iteration [default 1]
+
+
+While the bench program can be useful on its own, you'll probably want to run
+the benchmark with a bunch of different paramters. To do so automatically, you
+can use `runner.sh`. First, you'll need to create a file inside the `machines/`
+directory describing the machine you want to test on and its disks. The name of
+the file can be any name you want for the machine, and the file itself is
+interpreted by bash.
+
+Here's an example for testing against a single disk on localhost:
+
+    disks="somedisk:sda:/tmp"
+    rootcmd () {
+        sudo $*
+    }
+    cmd () {
+        $*
+    }
+    sendfile () {
+        cp $1 ~/
+    }
+
+And here's a more complex example for testing against two disks on a remote
+host:
+
+    disks="x25-m:sda:/home/ongardie 530:sdb:/home/ongardie/sdb1/tmp"
+    rootcmd () {
+        ssh flygecko sudo $*
+    }
+    cmd () {
+        ssh flygecko $*
+    }
+    sendfile () {
+        scp $1 flygecko:
+    }
+
+There's four things you need to define in each machine file:
+
+ * _disks_ is a string listing out the disks attached to the system. Each disk
+   is separated with a space. There are three components to each disk,
+   separated by colons: the disk's friendly name, the disk's device name within
+   `/dev/`, and the directory in which the benchmark should write its file.
+ * _rootcmd_ is a function that executes a command as root. This is used for
+   changing device settings with hdparm.
+ * _cmd_ is a function that executes a command as a normal user. This is used
+   for compiling and running the `bench` program.
+ * _sendfile_ is a function that copies the given file to the machine's home
+   directory.
+
+You'll need gcc and hdparm installed on every remote machine.
+
+Once you've got your machine description file in place, you should review the
+options at the top of the `runner.sh` script. It's rather thorough out of the
+box, and you might want to cut back on the number of configurations. When
+you're ready, run:
+
+    ./runner.sh
+
+This script produces a file called `results.csv` that looks something like
+this:
+
+    machine,disk,writecache,size,offset,direct,count,seconds
+    flygecko,x25-m,yes,1,0,no,100,0.033465235
+    flygecko,x25-m,yes,1,512,no,100,0.029445542
+    flygecko,x25-m,yes,2,0,no,100,0.034066043
+    flygecko,x25-m,yes,2,512,no,100,0.032349597
+    flygecko,x25-m,yes,4,0,no,100,0.030776787
+    flygecko,x25-m,yes,4,512,no,100,0.033931288
+    ...
+
+You'll probbaly want to `tail -f results.csv` while `runner.sh` is working so
+that you feel better about progress.
+
+Because the `results.csv` file gets to be large, it's probably necessary to
+post-process it somehow for human consumption. The script `post.R` loads in the
+`results.csv` file, summarizes the data a bit, and graphs the results. To
+invoke it, run:
+
+    R -e "source('post.R'); ggsave('results.svg', g, width=10, height=7)"
+
+If you know R and ggplot2, you'll find this to be a good starting point. If you
+don't, your mileage may vary with the default graph. A sample graph is included
+in `results.sample.svg`:
+
+![results.sample.svg](results.sample.svg).
+
+That's everything you'll find here for now. Go test your disks! Pull requests
+are welcome.

header.csv

@@ -0,0 +1 @@
+machine,disk,writecache,size,offset,direct,count,seconds

post.R

@@ -0,0 +1,57 @@
+# Copyright (c) 2014 Stanford University
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR(S) DISCLAIM ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL AUTHORS BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+require('plyr')
+require('ggplot2')
+require('scales')
+
+# turn raw number of bytes into human readable
+# will round if you don't pass in powers of two
+humanbytes <- function(x) {
+  ifelse(x < 1024,
+         x,
+         ifelse(x < 1024 * 1024,
+                sprintf('%0.0fK', x / 1024),
+                sprintf('%0.0fM', x / 1024 / 1024)))
+}
+
+# read results, take the minimum time of any offset and run
+results <- read.csv('results.csv')
+results$usperop <- results$seconds/results$count*1e6
+best <- ddply(results,
+              c('machine', 'disk', 'writecache', 'size', 'direct'),
+              summarize, usperop=min(usperop))
+
+# create plot
+g <- ggplot(best[best$direct=='no',], # direct=yes gets too crowded
+            aes(x=size,
+                y=usperop/1000,
+                color=disk,
+                group=interaction(machine, disk, writecache, direct))) +
+     geom_point(aes(shape=machine),
+                size=3) +
+     geom_line(aes(linetype=writecache), size=.75) +
+     coord_cartesian(xlim=(c(1, 2**21)*3/4),
+                     ylim=c(.05, 500)) +
+                     #linear: ylim=(c(0, 30000) + c(-1,1)*500)) +
+     scale_x_continuous(breaks=2**(0:20),
+                        trans=log_trans(2),
+                        labels=humanbytes) +
+     scale_y_continuous(breaks=c(.01, .1, 1, 10, 100),
+                        minor_breaks=NULL,
+                        trans=log_trans(10)) +
+     annotation_logticks(base=10, sides='l') +
+     #linear: scale_y_continuous(breaks=.1:10*3000) +
+     xlab('size per write (bytes)') +
+     ylab('time per write+fdatasync (milliseconds)')