Research performance and optimization

[nabeken]

We introduced delayd-bench tool.

I expect single node should handle at least 1000 msg/s because etcd establishes this throughput with raft.

raft requires many IOPS to handle transactions. Thus we perform tests on tmpfs and SSD-backed instance store for /tmp and /var/lib/docker for observing how a type of storage affects a result.

How to perform

$ fig up rabbitmq
$ delayd-bench -c delayd.toml -n 10000 -d 0

Just a quick note on c3.large

on SSD-backed instance-store with RAID0:

[INFO] delayd: sent 10000 messages for 1.186010864s, 8431.625969 msg/s
[INFO] delayd: processed 10000 messages for 16.03823705s, 623.509926 msg/s
[INFO] delayd: total: processed 10000 messages for 17.224247914s, 580.576873 msg/s

on tmpfs:

[INFO] delayd: sent 10000 messages for 1.338507415s, 7471.008295 msg/s
[INFO] delayd: processed 10000 messages for 5.73211508s, 1744.556740 msg/s
[INFO] delayd: total: processed 10000 messages for 7.070622495s, 1414.302631 msg/s

The performance on SSD-backed is not acceptable to us.

on Digital Ocean:

[INFO] delayd: sent 10000 messages for 1.53663287s, 6507.735319 msg/s
[INFO] delayd: processed 10000 messages for 23.643386653s, 422.951253 msg/s
[INFO] delayd: total: processed 10000 messages for 25.180019523s, 397.140280 msg/s

(on tmpfs)
[INFO] delayd: sent 10000 messages for 1.571370519s, 6363.871461 msg/s
[INFO] delayd: processed 10000 messages for 11.594584881s, 862.471585 msg/s
[INFO] delayd: total: processed 10000 messages for 13.1659554s, 759.534701 msg/s

okay, we need something different data store for raft log storage in the future.