This repo contains some of the tests I used to benchmark distributed storage solutions such as Portworx, Ceph and GlusterFS. The purpose of these tests is to evaluate the performance of a shared filesystem by simulating a near real-world workload:
- multiple hosts are connected to the same storage volume
- there are mixed VFS operations running at the same time: read, write, getattr, setattr, delete etc
- disk access is random
- the IO size is variable
- the files sizes are variable
- Any modern Linux distro on which you can install Java v1.7+
For running the tests, we will use 3 machines:
| Hostname | Role |
|---|---|
| master | Master node, the tests will be initiated from this node |
| worker1 | Worker node, this node will actually run the tests |
| worker2 | Worker node, this node will actually run the tests |
We assume the following:
- Java is installed on all machines and we have enough memory for the Vdbench's Java HEAP
- we export a shared fs volume and we mount it on both worker1 and worker2 as /vdbench
We setup passwordless SSH access between the master and the workers by running the following commands on the master node:
- Create SSH RSA key:
ssh-keygen -t rsa
- Copy the public key on the workers:
ssh-copy-id root@worker1
ssh-copy-id root@worker2
We clone this repository on each node in the same path, eg: /opt:
cd /opt
git clone https://github.com/iul1an/vdbench-tests.git
If required, we update the params file using this reference:
| Parameter | Description | Default | Obs. |
|---|---|---|---|
vdbench_path |
vdbench path on the worker nodes | $(pwd)/vdbench |
must be the same path on all nodes |
host1 |
first worker's hostname or IP address | worker1 | |
anchor1 |
the directory on which the test files are written on the first host | /vdbench/anchor1 | anchor1 and anchor2 must be different |
host2 |
second worker's hostname or IP address | worker2 | |
anchor2 |
the directory on which the test files are written on the first host | /vdbench/anchor2 | anchor1 and anchor2 must be different |
cd /opt/vdbench-tests
./run.sh
The results are in the "results" directory.
- Some of the tests are running using only one worker, worker1, because Vdbench doesn't allow you to use skew in multi-JVM mode
- There is a Vdbench manual in this repo which will help you modify the tests according to your needs, if you set a very large number of files in tests, make sure to also update the Java HEAP size in the vdbench script
- You can simulate a test without actually running it:
/opt/vdbench-tests/vdbench/vdbench -f testfile.vdb -s