Benchmark zfs send

[exarkun]

There is folk wisdom that the performance of zfs send is closely related to whether it is able to fill up its output buffer or not (if it is able to fill it then it performs poorly - particularly if emptying that buffer involved network round-trips).

HybridCluster suffered from performance problems until it added a buffering layer between zfs send and the network.

Measure the throughput of zfs send on a well-known dataset against various sizes of output buffer.

Generate machine-parseable output for consumption by a continuous benchmarking system (sadly yet-to-be-implemented).

[exarkun]

zfs send sounds concrete and precise but it's actually used as a placeholder for "the code that we will use to produce a snapshot data stream". This might involve calling into libzfs or libzfs_core APIs rather than running the zfs binary.

Figure out which of these approaches we're actually going to use before bothering to write this benchmark.

[exarkun]

Rob is particularly interested in having this benchmark done before the first release.

[robhaswell]

The motivation is that we experienced very poor performance for this operation in HybridCluster without a large buffer for ZFS to write to.

[itamarst]

http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ suggests the issue isn't zfs send, but rather zfs recv stalling the send pipeline.

[exarkun]

Thanks for tracking that down. The mismatch between the explanation there and the work description on this issue is a good argument for not spending time prematurely benchmarking anything (if someone had spent time benchmarking zfs send in isolation there's a good chance this would have been wasted effort because it would have missed out on the interaction with zfs recv).

Closing this in favor of benchmarking some software we actually build.

[exarkun]

See #70