Poor write speed over GlusterFS remote file system

[Ratio2]

Especially the problem when writing in small blocks. This is an important issue because some programs write in small blocks and this is not under our control.

If we take a sufficiently large 64k buffer (a fairly universal and reasonable choice, since it does not lead to a loss of speed in almost any case), then the speed drops by 3-4 times.

gocryptfs 2.4.0
gocryptfs -fg -notifypid=23471 /mnt/gluster/gocrypt /mnt/gluster-gocrypt -o rw,noprealloc,passfile=/home/duke/gocrypt-pwd.txt,allow_other,kernel_cache,dev,suid

glusterfs:

duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=4k count=32k
32768+0 records in
32768+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 27,5183 s, 4,9 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=8k count=16k
16384+0 records in
16384+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 27,5324 s, 4,9 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=16k count=8k
8192+0 records in
8192+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 27,7391 s, 4,8 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=32k count=4k
4096+0 records in
4096+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 26,7369 s, 5,0 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=64k count=2k
2048+0 records in
2048+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 28,7667 s, 4,7 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=128k count=1k
1024+0 records in
1024+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 30,4747 s, 4,4 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=256k count=512
512+0 records in
512+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 30,2351 s, 4,4 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=512k count=256
256+0 records in
256+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 27,4299 s, 4,9 MB/s
duke@doom:/mnt/gluster/tmp$ dd if=/dev/urandom of=blob bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 33,1746 s, 4,0 MB/s

gocryptfs over glusterfs:

duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=4k count=32k
32768+0 records in
32768+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 1360,89 s, 98,6 kB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=8k count=16k
16384+0 records in
16384+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 671,213 s, 200 kB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=16k count=8k
8192+0 records in
8192+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 344,592 s, 389 kB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=32k count=4k
4096+0 records in
4096+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 205,042 s, 655 kB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=64k count=2k
2048+0 records in
2048+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 104,363 s, 1,3 MB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=128k count=1k
1024+0 records in
1024+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 79,5411 s, 1,7 MB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=256k count=512
512+0 records in
512+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 62,9195 s, 2,1 MB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=512k count=256
256+0 records in
256+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 58,5022 s, 2,3 MB/s
duke@doom:/mnt/gluster-gocrypt/tmp$ dd if=/dev/urandom of=blob bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 54,9273 s, 2,4 MB/s

encfs over glusterfs:

duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=4k count=32k
32768+0 records in
32768+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 46,1873 s, 2,9 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=8k count=16k
16384+0 records in
16384+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 46,1054 s, 2,9 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=16k count=8k
8192+0 records in
8192+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 37,5365 s, 3,6 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=32k count=4k
4096+0 records in
4096+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 44,2531 s, 3,0 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=64k count=2k
2048+0 records in
2048+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 41,1942 s, 3,3 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=128k count=1k
1024+0 records in
1024+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 43,1311 s, 3,1 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=256k count=512
512+0 records in
512+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 42,8404 s, 3,1 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=512k count=256
256+0 records in
256+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 41,1991 s, 3,3 MB/s
duke@doom:/mnt/tmp$ dd if=/dev/urandom of=blob bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (134 MB, 128 MiB) copied, 45,4358 s, 3,0 MB/s

[rfjakob]

Thank you for the detailed report. The comparison with encfs is most helpful. Especially at 4k, looks like gocryptfs could do much better. I will check what the difference is.