Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incoherency between throughput displayed by cgiserv and collectd/bwm-ng #864

Open
baby-gnu opened this issue Feb 5, 2020 · 2 comments
Open

Comments

@baby-gnu
Copy link

@baby-gnu baby-gnu commented Feb 5, 2020

Hello.

To discover why Lizardfs is so slow on my 10 SSD, I'm looking at some monitoring.

In addition to the cgiserv provided by Lizardfs, I setup a collectd (with graphite and grafana on a dedicated computer) and I'm looking at the output of bwm-ng to get some live monitoring.

I have incoherent results between the disk throughput shown by cgiserv and the stats I got with collectd and bwm-ng.

As you will see below:

  • I only tested an standalone server to measure the stripping perf of Lizardfs without any replication
  • I modified the Storage Tuner Benchmark to report result against a single SSD access
    diff --git a/storage-tuner-benchmark b/storage-tuner-benchmark
    index 3f60413..0b0617a 100755
    --- a/storage-tuner-benchmark
    +++ b/storage-tuner-benchmark
    @@ -174,9 +174,42 @@ Testgroup "current"
     16 files, 16 threads each, rnd 16k reads, native aio: 494 read iops (123% HDD)
     HEREDOC
     )
    -	
    +
    +	# HGST Ultrastar SS300 HUSMR3280ASS200 (800GB SAS 12GB/s)
     	myBaselineSSD=$(cat <<'HEREDOC'
    -TODO
    +storage-tuner-benchmark version 2.1.0
    +Testgroup "current"
    +===   1 file series   ===
    +1 file, 1 thread, seq 1M writes, simple: 672 write iops (371% HDD)
    +1 file, 1 thread, rnd 16k writes, simple: 19084 write iops (2680% HDD)
    +1 file, 1 thread, rnd 16k writes, simple, take 2: 19215 write iops (2920% HDD)
    +1 file, 16 threads, rnd 4k writes, posixaio: 66057 write iops (18819% HDD)
    +1 file, 16 threads, rnd 8k writes, posixaio: 46166 write iops (10888% HDD)
    +1 file, 16 threads, rnd 16k writes, posixaio: 28990 write iops (6485% HDD)
    +1 file, 16 threads, rnd 16k writes, posixaio, take 2: 29104 write iops (5774% HDD)
    +===  16 file series   ===
    +16 files, 1 thread each, seq 1M writes, simple: 994 write iops (662% HDD)
    +16 files, 1 thread each, rnd 16k writes, simple: 55165 write iops (8435% HDD)
    +16 files, 1 thread each, rnd 16k writes, simple, take 2: 55277 write iops (8816% HDD)
    +16 files, 1 thread each, rnd 16k writes, posixaio: 53785 write iops (11541% HDD)
    +16 files, 16 threads each, rnd 16k writes, posixaio: 54710 write iops (10188% HDD)
    +16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 54462 write iops (10870% HDD)
    +===   O_SYNC series   ===
    +1 file, 1 thread, rnd 16k writes, simple, o_sync: 18798 write iops (12127% HDD)
    +1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 18012 write iops (7473% HDD)
    +16 files, 1 thread each, rnd 16k writes, simple, o_sync: 53479 write iops (10589% HDD)
    +16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 53158 write iops (10443% HDD)
    +===    read series    ===
    +1 file, 1 thread, seq 1M reads, simple: 781 read iops (20% HDD)
    +1 file, 16 threads, rnd 16k reads, posixaio: 5142 read iops (3007% HDD)
    +16 files, 1 thread each, seq 1M reads, simple: 1077 read iops (26% HDD)
    +16 files, 1 thread each, rnd 16k reads, posixaio: 48398 read iops (10297% HDD)
    +16 files, 16 threads each, rnd 16k reads, posixaio: 48936 read iops (9866% HDD)
    +=== native aio series ===
    +1 file, 16 threads, rnd 16k writes, native aio: 18903 write iops (3083% HDD)
    +16 files, 16 threads each, rnd 16k writes, native aio: 55334 write iops (11044% HDD)
    +1 file, 16 threads, rnd 16k reads, native aio: 5062 read iops (3864% HDD)
    +16 files, 16 threads each, rnd 16k reads, native aio: 49661 read iops (10052% HDD)
     HEREDOC
     )
  • I made the same tests against a software RAID0 (mdadm)

Setup

I'm testing a new infrastructure composed of 5 servers, each one is:

  • Debian Buster
  • 72 cores Xeon(R) Gold 5220 CPU @ 2.20GHz
  • 396GB of RAM
  • 10 * 800GB SSD SAS 12GB/s
  • Kernel 4.19.0-6-amd64
  • Lizardfs 3.12.0+dfsg-3+b10 from distribution

Tests

I tried several things to test the IO:

  • put 335GB of ISO DVD in a tmpfs (under /mnt/ramfs/isos) and
    • copy the source directory to the mount point
    • copy the source directory using parallel with 20 concurrent jobs
    • rsync the source directory to the mount point
  • run storage-tuner-benchmark
  • run bonnie++
    • a single process
    bonnie++ -d /mnt/test/ -u bonnie -m $(hostname)-lizardfs -n 64:100000:16:64 -b -q > /root/lizardfs/bonnie++-single 2>&1
    
    • 10 processes
    bonnie++ -d /mnt/test -u bonnie -p10
    
    for i in {1..10}
    do
        (nohup bonnie++ -ys -d /mnt/test/ -u bonnie -m $(hostname)-lizardfs-${i} -r 24g -s 400G -n 64:100000:16:64 -b -q > /root/lizardfs/bonnie++-parallel-${i} 2>&1 & )
    done
    

To get statistics:

  • I installed collectd (5 seconds measurement interval) and send everything to a dedicated machine with graphite and grafana. This is very light on each server for a very good accuracy.
  • I installed bwm-ng (bwm-ng -i disk -T avg -d 1 -t 2000)

Lizardfs stripping perf (standalone server without any replication)

Now, I'm testing without any replication just to see how lizardfs perform locally, on each server I have:

  • 10 SSD without RAID formatted as XFS
    • each one has /etc/fstab options rw,noexec,nodev,noatime,nodiratime,noquota,largeio,inode64,barrier=0
    • when mounted, each one has /proc/mounts options rw,relatime,attr2,inode64,noquota (other options are not in xfs(5))
  • a local master
  • a local chunkserver connected to the local master and using the 10 SSD (so stripping data between each one if I understand correctly)

So each server has a lizardfs of 7.2TB available space used only locally with the mount point /mnt/test:

  • /etc/fstab entry
    mfsmount		/mnt/test	fuse	big_writes,cacheexpirationtime=0,mfsdelayedinit,mfsmaster=127.0.0.1,_netdev	0 0
    
  • /proc/mounts (big_writes has no effect? It's not in mount.fuse(8) manpage)
    mfs#127.0.0.1:9421 /mnt/test fuse rw,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
    

Copy of 335GB from RAM

I just run cp -r /mnt/ramfs/isos /mnt/test/:

  • cgiserv: around 900MiB/s on each SSD
  • collectd: around 40MB/s on each SSD (60 to 90 IO/s per SSD in 5 minutes average)
  • bwm-ng: between 30MB/s to 40MB/s on each SSD

According to cgiserv the copy should take 40 seconds (335GB at 9GB/s) but it took 16 minutes (350MB/s which is 5 times slower than RAID0).

Copy of 335GB from RAM with 20 parallel jobs

I run find . | parallel -j 20 cp {} /mnt/test/isos/

  • cgiserv: around 1.1GiB/s on each SSD
  • collectd: around 130MB/s

According to cgiserv the copy should take 30 seconds (335GB at 11GB/s) but it took 4 minutes 20 seconds (1.29GB/s which is 6 to 8 times slower than RAID0).

rsync of 335GB from RAM

I run rysnc -rvWP /mnt/ramfs/isos/ /mnt/test/isos/:

  • cgiserv: around 900MB/s on each SSD
  • collectd: around 25MB/s on each SSD (50 IO/s in 5 minutes average)
  • bwm-ng: around 25MB/s on each SSD

According to cgiserv the rsync should take 40 seconds (335GB at 9GB/s) but it took 26 minutes (214MB/s which is 2 times slower than RAID0).

Here is the output of rsync itself:

sending incremental file list
created directory /mnt/test/isos
./
dvd1.iso

  5,441,959,936 100%  201.73MB/s    0:00:25 (xfr#1, to-chk=65/67)
dvd10.iso

  5,441,959,936 100%  206.52MB/s    0:00:25 (xfr#2, to-chk=64/67)
dvd11.iso

  5,441,959,936 100%  220.23MB/s    0:00:23 (xfr#3, to-chk=63/67)
dvd12.iso

  5,441,959,936 100%  215.45MB/s    0:00:24 (xfr#4, to-chk=62/67)
dvd13.iso

  5,441,959,936 100%  222.06MB/s    0:00:23 (xfr#5, to-chk=61/67)
dvd14.iso

  5,441,959,936 100%  228.87MB/s    0:00:22 (xfr#6, to-chk=60/67)
dvd15.iso

  5,441,959,936 100%  257.97MB/s    0:00:20 (xfr#7, to-chk=59/67)
dvd16.iso

  5,441,959,936 100%  219.18MB/s    0:00:23 (xfr#8, to-chk=58/67)
dvd17.iso

  5,441,959,936 100%  210.99MB/s    0:00:24 (xfr#9, to-chk=57/67)
dvd18.iso

  5,441,959,936 100%  212.12MB/s    0:00:24 (xfr#10, to-chk=56/67)
dvd19.iso

  5,441,959,936 100%  209.28MB/s    0:00:24 (xfr#11, to-chk=55/67)
dvd2.iso

  5,441,959,936 100%  218.19MB/s    0:00:23 (xfr#12, to-chk=54/67)
dvd20.iso

  5,441,959,936 100%  207.30MB/s    0:00:25 (xfr#13, to-chk=53/67)
dvd21.iso

  5,441,959,936 100%  222.25MB/s    0:00:23 (xfr#14, to-chk=52/67)
dvd22.iso

  5,441,959,936 100%  218.25MB/s    0:00:23 (xfr#15, to-chk=51/67)
dvd23.iso

  5,441,959,936 100%  214.62MB/s    0:00:24 (xfr#16, to-chk=50/67)
dvd24.iso

  5,441,959,936 100%  215.16MB/s    0:00:24 (xfr#17, to-chk=49/67)
dvd25.iso

  5,441,959,936 100%  227.52MB/s    0:00:22 (xfr#18, to-chk=48/67)
dvd26.iso

  5,441,959,936 100%  207.30MB/s    0:00:25 (xfr#19, to-chk=47/67)
dvd27.iso

  5,441,959,936 100%  227.74MB/s    0:00:22 (xfr#20, to-chk=46/67)
dvd28.iso

  5,441,959,936 100%  208.60MB/s    0:00:24 (xfr#21, to-chk=45/67)
dvd29.iso

  5,441,959,936 100%  205.47MB/s    0:00:25 (xfr#22, to-chk=44/67)
dvd3.iso

  5,441,959,936 100%  212.07MB/s    0:00:24 (xfr#23, to-chk=43/67)
dvd30.iso

  5,441,959,936 100%  216.21MB/s    0:00:24 (xfr#24, to-chk=42/67)
dvd31.iso

  5,441,959,936 100%  230.70MB/s    0:00:22 (xfr#25, to-chk=41/67)
dvd32.iso

  5,441,959,936 100%  217.72MB/s    0:00:23 (xfr#26, to-chk=40/67)
dvd33.iso

  5,441,959,936 100%  207.39MB/s    0:00:25 (xfr#27, to-chk=39/67)
dvd34.iso

  5,441,959,936 100%  231.04MB/s    0:00:22 (xfr#28, to-chk=38/67)
dvd35.iso

  5,441,959,936 100%  220.92MB/s    0:00:23 (xfr#29, to-chk=37/67)
dvd36.iso

  5,441,959,936 100%  204.09MB/s    0:00:25 (xfr#30, to-chk=36/67)
dvd37.iso

  5,441,959,936 100%  206.99MB/s    0:00:25 (xfr#31, to-chk=35/67)
dvd38.iso

  5,441,959,936 100%  215.03MB/s    0:00:24 (xfr#32, to-chk=34/67)
dvd39.iso

  5,441,959,936 100%  229.02MB/s    0:00:22 (xfr#33, to-chk=33/67)
dvd4.iso

  5,441,959,936 100%  241.97MB/s    0:00:21 (xfr#34, to-chk=32/67)
dvd40.iso

  5,441,959,936 100%  217.49MB/s    0:00:23 (xfr#35, to-chk=31/67)
dvd41.iso

  5,441,959,936 100%  219.97MB/s    0:00:23 (xfr#36, to-chk=30/67)
dvd42.iso

  5,441,959,936 100%  216.25MB/s    0:00:23 (xfr#37, to-chk=29/67)
dvd43.iso

  5,441,959,936 100%  235.31MB/s    0:00:22 (xfr#38, to-chk=28/67)
dvd44.iso

  5,441,959,936 100%  233.85MB/s    0:00:22 (xfr#39, to-chk=27/67)
dvd45.iso

  5,441,959,936 100%  214.33MB/s    0:00:24 (xfr#40, to-chk=26/67)
dvd46.iso

  5,441,959,936 100%  208.43MB/s    0:00:24 (xfr#41, to-chk=25/67)
dvd47.iso

  5,441,959,936 100%  207.77MB/s    0:00:24 (xfr#42, to-chk=24/67)
dvd48.iso

  5,441,959,936 100%  203.35MB/s    0:00:25 (xfr#43, to-chk=23/67)
dvd49.iso

  5,441,959,936 100%  210.28MB/s    0:00:24 (xfr#44, to-chk=22/67)
dvd5.iso

  5,441,959,936 100%  214.17MB/s    0:00:24 (xfr#45, to-chk=21/67)
dvd50.iso

  5,441,959,936 100%  221.94MB/s    0:00:23 (xfr#46, to-chk=20/67)
dvd51.iso

  5,441,959,936 100%  218.02MB/s    0:00:23 (xfr#47, to-chk=19/67)
dvd52.iso

  5,441,959,936 100%  218.07MB/s    0:00:23 (xfr#48, to-chk=18/67)
dvd53.iso

  5,441,959,936 100%  223.86MB/s    0:00:23 (xfr#49, to-chk=17/67)
dvd54.iso

  5,441,959,936 100%  215.84MB/s    0:00:24 (xfr#50, to-chk=16/67)
dvd55.iso

  5,441,959,936 100%  213.72MB/s    0:00:24 (xfr#51, to-chk=15/67)
dvd56.iso

  5,441,959,936 100%  212.62MB/s    0:00:24 (xfr#52, to-chk=14/67)
dvd57.iso

  5,441,959,936 100%  214.78MB/s    0:00:24 (xfr#53, to-chk=13/67)
dvd58.iso

  5,441,959,936 100%  213.39MB/s    0:00:24 (xfr#54, to-chk=12/67)
dvd59.iso

  5,441,959,936 100%  217.41MB/s    0:00:23 (xfr#55, to-chk=11/67)
dvd6.iso

  5,441,959,936 100%  202.61MB/s    0:00:25 (xfr#56, to-chk=10/67)
dvd60.iso

  5,441,959,936 100%  201.11MB/s    0:00:25 (xfr#57, to-chk=9/67)
dvd61.iso

  5,441,959,936 100%  198.87MB/s    0:00:26 (xfr#58, to-chk=8/67)
dvd62.iso

  5,441,959,936 100%  209.06MB/s    0:00:24 (xfr#59, to-chk=7/67)
dvd63.iso

  5,441,959,936 100%  202.95MB/s    0:00:25 (xfr#60, to-chk=6/67)
dvd64.iso

  5,441,959,936 100%  219.84MB/s    0:00:23 (xfr#61, to-chk=5/67)
dvd65.iso

  5,441,959,936 100%  212.42MB/s    0:00:24 (xfr#62, to-chk=4/67)
dvd66.iso

  5,441,959,936 100%  211.74MB/s    0:00:24 (xfr#63, to-chk=3/67)
dvd7.iso

  5,441,959,936 100%  207.44MB/s    0:00:25 (xfr#64, to-chk=2/67)
dvd8.iso

  5,441,959,936 100%  213.68MB/s    0:00:24 (xfr#65, to-chk=1/67)
dvd9.iso

  5,441,959,936 100%  217.59MB/s    0:00:23 (xfr#66, to-chk=0/67)

sent 359,257,047,755 bytes  received 1,310 bytes  230,366,815.69 bytes/sec
total size is 359,169,355,776  speedup is 1.00

Storage Tuner Benchmark

No need to compare against RAID0 with the SSD percentage initialised with one of the disk:

Creating test directory "stb-testdir"
Running tests in "/mnt/test/stb-testdir" on nebula80 @ 2020-02-04 14:58:12 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 371 write iops (204% HDD / 55% SSD)
1 file, 1 thread, rnd 16k writes, simple: 1159 write iops (162% HDD / 6% SSD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 1197 write iops (181% HDD / 6% SSD)
1 file, 16 threads, rnd 4k writes, posixaio: 1786 write iops (508% HDD / 2% SSD)
1 file, 16 threads, rnd 8k writes, posixaio: 1792 write iops (422% HDD / 3% SSD)
1 file, 16 threads, rnd 16k writes, posixaio: 1774 write iops (396% HDD / 6% SSD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 1786 write iops (354% HDD / 6% SSD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 1145 write iops (763% HDD / 115% SSD)
16 files, 1 thread each, rnd 16k writes, simple: 15893 write iops (2430% HDD / 28% SSD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 15740 write iops (2510% HDD / 28% SSD)
16 files, 1 thread each, rnd 16k writes, posixaio: 15811 write iops (3392% HDD / 29% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio: 53541 write iops (9970% HDD / 97% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 51728 write iops (10324% HDD / 94% SSD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 772 write iops (498% HDD / 4% SSD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 829 write iops (343% HDD / 4% SSD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 15956 write iops (3159% HDD / 29% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 15625 write iops (3069% HDD / 29% SSD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 294 read iops (7% HDD / 37% SSD)
1 file, 16 threads, rnd 16k reads, posixaio: 1992 read iops (1164% HDD / 38% SSD)
16 files, 1 thread each, seq 1M reads, simple: 3779 read iops (93% HDD / 350% SSD)
16 files, 1 thread each, rnd 16k reads, posixaio: 55924 read iops (11898% HDD / 115% SSD)
16 files, 16 threads each, rnd 16k reads, posixaio: 56470 read iops (11385% HDD / 115% SSD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 820 write iops (133% HDD / 4% SSD)
16 files, 16 threads each, rnd 16k writes, native aio: 16161 write iops (3225% HDD / 29% SSD)
1 file, 16 threads, rnd 16k reads, native aio: 1633 read iops (1246% HDD / 32% SSD)
16 files, 16 threads each, rnd 16k reads, native aio: 55845 read iops (11304% HDD / 112% SSD)
Tests complete on nebula80 @ 2020-02-04 15:00:30.
Files remain. To clean up, add argument "cleanup".
Cleanup
Done.

Bonnie++

It takes ages to complete, during the running, at one point in time I had:

  • cgiserv: around 900MB/s for each SSD
  • collectd: around 8MB/s for each SSD
  • bwm-ng: around 7MB/s for each SSD

bonnie++ finally failed after hours (more than 10hours) with the following output:

Bonnie: drastic I/O error (rmdir): Directory not empty
Cleaning up test directory after error.

Raw perf without Lizardfs

To make a comparison, I started with a software RAID0 with mdadm using all the 10 SSD.

The monitoring is against the RAID0 device /dev/md0.

Copy of 335GB from RAM

I just run cp -r /mnt/ramfs/isos /mnt/test/, collectd tells me:

⮕ around 2GB/s
⮕ between 3.5k IOPS to 4.5k IOPS

Which is coherent with the output of bwm-ng.

Copy of 335GB from RAM with 20 parallel jobs

I run find . | parallel -j 20 cp {} /mnt/test/isos/, collectd tells me:

⮕ between 6GB/s to 8GB/s
⮕ around 20k IOPS

It's consistent with the output of bwm-ng

rsync of 335GB from RAM

rsync is quite slow in itself compared to cp, collectd tells me:

⮕ around 500MB/s
⮕ between 700 IOPS to 1.2k IOPS

Here is the rsync output itself:

sending incremental file list
created directory /mnt/test/isos
./
dvd1.iso

  5,441,959,936 100%  502.16MB/s    0:00:10 (xfr#1, to-chk=65/67)
dvd10.iso

  5,441,959,936 100%  473.96MB/s    0:00:10 (xfr#2, to-chk=64/67)
dvd11.iso

  5,441,959,936 100%  455.45MB/s    0:00:11 (xfr#3, to-chk=63/67)
dvd12.iso

  5,441,959,936 100%  474.22MB/s    0:00:10 (xfr#4, to-chk=62/67)
dvd13.iso

  5,441,959,936 100%  441.69MB/s    0:00:11 (xfr#5, to-chk=61/67)
dvd14.iso

  5,441,959,936 100%  451.84MB/s    0:00:11 (xfr#6, to-chk=60/67)
dvd15.iso

  5,441,959,936 100%  465.75MB/s    0:00:11 (xfr#7, to-chk=59/67)
dvd16.iso

  5,441,959,936 100%  484.31MB/s    0:00:10 (xfr#8, to-chk=58/67)
dvd17.iso

  5,441,959,936 100%  461.44MB/s    0:00:11 (xfr#9, to-chk=57/67)
dvd18.iso

  5,441,959,936 100%  484.40MB/s    0:00:10 (xfr#10, to-chk=56/67)
dvd19.iso

  5,441,959,936 100%  459.40MB/s    0:00:11 (xfr#11, to-chk=55/67)
dvd2.iso

  5,441,959,936 100%  470.01MB/s    0:00:11 (xfr#12, to-chk=54/67)
dvd20.iso

  5,441,959,936 100%  489.01MB/s    0:00:10 (xfr#13, to-chk=53/67)
dvd21.iso

  5,441,959,936 100%  463.09MB/s    0:00:11 (xfr#14, to-chk=52/67)
dvd22.iso

  5,441,959,936 100%  478.02MB/s    0:00:10 (xfr#15, to-chk=51/67)
dvd23.iso

  5,441,959,936 100%  451.53MB/s    0:00:11 (xfr#16, to-chk=50/67)
dvd24.iso

  5,441,959,936 100%  475.31MB/s    0:00:10 (xfr#17, to-chk=49/67)
dvd25.iso

  5,441,959,936 100%  456.77MB/s    0:00:11 (xfr#18, to-chk=48/67)
dvd26.iso

  5,441,959,936 100%  480.41MB/s    0:00:10 (xfr#19, to-chk=47/67)
dvd27.iso

  5,441,959,936 100%  457.05MB/s    0:00:11 (xfr#20, to-chk=46/67)
dvd28.iso

  5,441,959,936 100%  472.54MB/s    0:00:10 (xfr#21, to-chk=45/67)
dvd29.iso

  5,441,959,936 100%  449.11MB/s    0:00:11 (xfr#22, to-chk=44/67)
dvd3.iso

  5,441,959,936 100%  457.30MB/s    0:00:11 (xfr#23, to-chk=43/67)
dvd30.iso

  5,441,959,936 100%  473.96MB/s    0:00:10 (xfr#24, to-chk=42/67)
dvd31.iso

  5,441,959,936 100%  455.17MB/s    0:00:11 (xfr#25, to-chk=41/67)
dvd32.iso

  5,441,959,936 100%  479.61MB/s    0:00:10 (xfr#26, to-chk=40/67)
dvd33.iso

  5,441,959,936 100%  464.04MB/s    0:00:11 (xfr#27, to-chk=39/67)
dvd34.iso

  5,441,959,936 100%  487.17MB/s    0:00:10 (xfr#28, to-chk=38/67)
dvd35.iso

  5,441,959,936 100%  463.13MB/s    0:00:11 (xfr#29, to-chk=37/67)
dvd36.iso

  5,441,959,936 100%  481.79MB/s    0:00:10 (xfr#30, to-chk=36/67)
dvd37.iso

  5,441,959,936 100%  460.79MB/s    0:00:11 (xfr#31, to-chk=35/67)
dvd38.iso

  5,441,959,936 100%  483.32MB/s    0:00:10 (xfr#32, to-chk=34/67)
dvd39.iso

  5,441,959,936 100%  456.29MB/s    0:00:11 (xfr#33, to-chk=33/67)
dvd4.iso

  5,441,959,936 100%  461.65MB/s    0:00:11 (xfr#34, to-chk=32/67)
dvd40.iso

  5,441,959,936 100%  486.35MB/s    0:00:10 (xfr#35, to-chk=31/67)
dvd41.iso

  5,441,959,936 100%  461.57MB/s    0:00:11 (xfr#36, to-chk=30/67)
dvd42.iso

  5,441,959,936 100%  484.76MB/s    0:00:10 (xfr#37, to-chk=29/67)
dvd43.iso

  5,441,959,936 100%  465.54MB/s    0:00:11 (xfr#38, to-chk=28/67)
dvd44.iso

  5,441,959,936 100%  475.04MB/s    0:00:10 (xfr#39, to-chk=27/67)
dvd45.iso

  5,441,959,936 100%  445.75MB/s    0:00:11 (xfr#40, to-chk=26/67)
dvd46.iso

  5,441,959,936 100%  463.38MB/s    0:00:11 (xfr#41, to-chk=25/67)
dvd47.iso

  5,441,959,936 100%  474.31MB/s    0:00:10 (xfr#42, to-chk=24/67)
dvd48.iso

  5,441,959,936 100%  446.90MB/s    0:00:11 (xfr#43, to-chk=23/67)
dvd49.iso

  5,441,959,936 100%  464.67MB/s    0:00:11 (xfr#44, to-chk=22/67)
dvd5.iso

  5,441,959,936 100%  485.67MB/s    0:00:10 (xfr#45, to-chk=21/67)
dvd50.iso

  5,441,959,936 100%  458.51MB/s    0:00:11 (xfr#46, to-chk=20/67)
dvd51.iso

  5,441,959,936 100%  481.43MB/s    0:00:10 (xfr#47, to-chk=19/67)
dvd52.iso

  5,441,959,936 100%  457.05MB/s    0:00:11 (xfr#48, to-chk=18/67)
dvd53.iso

  5,441,959,936 100%  476.57MB/s    0:00:10 (xfr#49, to-chk=17/67)
dvd54.iso

  5,441,959,936 100%  452.79MB/s    0:00:11 (xfr#50, to-chk=16/67)
dvd55.iso

  5,441,959,936 100%  467.30MB/s    0:00:11 (xfr#51, to-chk=15/67)
dvd56.iso

  5,441,959,936 100%  473.01MB/s    0:00:10 (xfr#52, to-chk=14/67)
dvd57.iso

  5,441,959,936 100%  442.86MB/s    0:00:11 (xfr#53, to-chk=13/67)
dvd58.iso

  5,441,959,936 100%  453.78MB/s    0:00:11 (xfr#54, to-chk=12/67)
dvd59.iso

  5,441,959,936 100%  464.83MB/s    0:00:11 (xfr#55, to-chk=11/67)
dvd6.iso

  5,441,959,936 100%  473.31MB/s    0:00:10 (xfr#56, to-chk=10/67)
dvd60.iso

  5,441,959,936 100%  444.45MB/s    0:00:11 (xfr#57, to-chk=9/67)
dvd61.iso

  5,441,959,936 100%  454.81MB/s    0:00:11 (xfr#58, to-chk=8/67)
dvd62.iso

  5,441,959,936 100%  463.50MB/s    0:00:11 (xfr#59, to-chk=7/67)
dvd63.iso

  5,441,959,936 100%  474.00MB/s    0:00:10 (xfr#60, to-chk=6/67)
dvd64.iso

  5,441,959,936 100%  443.84MB/s    0:00:11 (xfr#61, to-chk=5/67)
dvd65.iso

  5,441,959,936 100%  455.13MB/s    0:00:11 (xfr#62, to-chk=4/67)
dvd66.iso

  5,441,959,936 100%  465.79MB/s    0:00:11 (xfr#63, to-chk=3/67)
dvd7.iso

  5,441,959,936 100%  481.48MB/s    0:00:10 (xfr#64, to-chk=2/67)
dvd8.iso

  5,441,959,936 100%  456.61MB/s    0:00:11 (xfr#65, to-chk=1/67)
dvd9.iso

  5,441,959,936 100%  478.33MB/s    0:00:10 (xfr#66, to-chk=0/67)

sent 359,257,047,755 bytes  received 1,310 bytes  513,591,206.67 bytes/sec
total size is 359,169,355,776  speedup is 1.00

Storage Tuner Benchmark

Creating test directory "stb-testdir"
Running tests in "/mnt/test/stb-testdir" on nebula80 @ 2020-02-04 08:46:58 ...
storage-tuner-benchmark version 2.1.0
Testgroup "current"
===   1 file series   ===
1 file, 1 thread, seq 1M writes, simple: 1426 write iops (787% HDD / 212% SSD)
1 file, 1 thread, rnd 16k writes, simple: 17856 write iops (2507% HDD / 93% SSD)
1 file, 1 thread, rnd 16k writes, simple, take 2: 17927 write iops (2724% HDD / 93% SSD)
1 file, 16 threads, rnd 4k writes, posixaio: 61295 write iops (17462% HDD / 92% SSD)
1 file, 16 threads, rnd 8k writes, posixaio: 51210 write iops (12077% HDD / 110% SSD)
1 file, 16 threads, rnd 16k writes, posixaio: 36122 write iops (8080% HDD / 124% SSD)
1 file, 16 threads, rnd 16k writes, posixaio, take 2: 36721 write iops (7285% HDD / 126% SSD)
===  16 file series   ===
16 files, 1 thread each, seq 1M writes, simple: 4830 write iops (3220% HDD / 485% SSD)
16 files, 1 thread each, rnd 16k writes, simple: 179819 write iops (27495% HDD / 325% SSD)
16 files, 1 thread each, rnd 16k writes, simple, take 2: 180410 write iops (28773% HDD / 326% SSD)
16 files, 1 thread each, rnd 16k writes, posixaio: 146310 write iops (31396% HDD / 272% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio: 309495 write iops (57634% HDD / 565% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 310399 write iops (61955% HDD / 569% SSD)
===   O_SYNC series   ===
1 file, 1 thread, rnd 16k writes, simple, o_sync: 16672 write iops (10756% HDD / 88% SSD)
1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 16675 write iops (6919% HDD / 92% SSD)
16 files, 1 thread each, rnd 16k writes, simple, o_sync: 175915 write iops (34834% HDD / 328% SSD)
16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 177008 write iops (34775% HDD / 332% SSD)
===    read series    ===
1 file, 1 thread, seq 1M reads, simple: 2900 read iops (77% HDD / 371% SSD)
1 file, 16 threads, rnd 16k reads, posixaio: 5105 read iops (2985% HDD / 99% SSD)
16 files, 1 thread each, seq 1M reads, simple: 5797 read iops (143% HDD / 538% SSD)
16 files, 1 thread each, rnd 16k reads, posixaio: 66565 read iops (14162% HDD / 137% SSD)
16 files, 16 threads each, rnd 16k reads, posixaio: 69165 read iops (13944% HDD / 141% SSD)
=== native aio series ===
1 file, 16 threads, rnd 16k writes, native aio: 17903 write iops (2920% HDD / 94% SSD)
16 files, 16 threads each, rnd 16k writes, native aio: 178593 write iops (35647% HDD / 322% SSD)
1 file, 16 threads, rnd 16k reads, native aio: 5023 read iops (3834% HDD / 99% SSD)
16 files, 16 threads each, rnd 16k reads, native aio: 70406 read iops (14252% HDD / 141% SSD)
Tests complete on nebula80 @ 2020-02-04 08:49:15.
Files remain. To clean up, add argument "cleanup".
Cleanup
Done.

Bonnie++

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
nebula80-raid0 753G 2694k  99  1.7g  99  1.6g  91 4665k  99  3.0g  99 +++++ +++
Latency              6370us    1943us   67110us    2430us    5762us    1696us
Version  1.98       ------Sequential Create------ --------Random Create--------
nebula80-raid0      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
    64:100000:16/64 65536  32 65536  99 65536  46 65536  32 65536  99 65536  46
Latency              5662us    1528us   62180us    2861us    1628us   77778us
@baby-gnu

This comment has been minimized.

Copy link
Author

@baby-gnu baby-gnu commented Feb 5, 2020

I installed tuned and activated network-throughput profile and things are getting a little bit better:

  • simple cp is now 40% of RAID0 (805MB/s instead of 2GB/s)
  • 20 parallel cp is 24.5% of RAID0 (1.735GB/s instead of ~7GB/s)
  • rsync is 83.4% of RAID0 (292MB/s instead of 350MB/s)
  • Storage Tuner Benchmark is a little bit better but not that much
    Running tests in "/mnt/test/stb-testdir" on nebula81 @ 2020-02-05 11:32:45 ...
    storage-tuner-benchmark version 2.1.0
    Testgroup "current"
    ===   1 file series   ===
    1 file, 1 thread, seq 1M writes, simple: 885 write iops (488% HDD / 131% SSD)
    1 file, 1 thread, rnd 16k writes, simple: 3180 write iops (446% HDD / 16% SSD)
    1 file, 1 thread, rnd 16k writes, simple, take 2: 3075 write iops (467% HDD / 16% SSD)
    1 file, 16 threads, rnd 4k writes, posixaio: 4405 write iops (1254% HDD / 6% SSD)
    1 file, 16 threads, rnd 8k writes, posixaio: 4656 write iops (1098% HDD / 10% SSD)
    1 file, 16 threads, rnd 16k writes, posixaio: 4871 write iops (1089% HDD / 16% SSD)
    1 file, 16 threads, rnd 16k writes, posixaio, take 2: 4113 write iops (816% HDD / 14% SSD)
    ===  16 file series   ===
    16 files, 1 thread each, seq 1M writes, simple: 1753 write iops (1168% HDD / 176% SSD)
    16 files, 1 thread each, rnd 16k writes, simple: 27577 write iops (4216% HDD / 49% SSD)
    16 files, 1 thread each, rnd 16k writes, simple, take 2: 27659 write iops (4411% HDD / 50% SSD)
    16 files, 1 thread each, rnd 16k writes, posixaio: 27501 write iops (5901% HDD / 51% SSD)
    16 files, 16 threads each, rnd 16k writes, posixaio: 61761 write iops (11501% HDD / 112% SSD)
    16 files, 16 threads each, rnd 16k writes, posixaio, take 2: 62114 write iops (12398% HDD / 114% SSD)
    ===   O_SYNC series   ===
    1 file, 1 thread, rnd 16k writes, simple, o_sync: 2302 write iops (1485% HDD / 12% SSD)
    1 file, 16 threads, rnd 16k writes, posixaio, o_sync: 3326 write iops (1380% HDD / 18% SSD)
    16 files, 1 thread each, rnd 16k writes, simple, o_sync: 27608 write iops (5466% HDD / 51% SSD)
    16 files, 16 threads each, rnd 16k writes, posixaio, o_sync: 27310 write iops (5365% HDD / 51% SSD)
    ===    read series    ===
    1 file, 1 thread, seq 1M reads, simple: 502 read iops (13% HDD / 64% SSD)
    1 file, 16 threads, rnd 16k reads, posixaio: 4097 read iops (2395% HDD / 79% SSD)
    16 files, 1 thread each, seq 1M reads, simple: 3873 read iops (95% HDD / 359% SSD)
    16 files, 1 thread each, rnd 16k reads, posixaio: 56729 read iops (12070% HDD / 117% SSD)
    16 files, 16 threads each, rnd 16k reads, posixaio: 57083 read iops (11508% HDD / 116% SSD)
    === native aio series ===
    1 file, 16 threads, rnd 16k writes, native aio: 2618 write iops (427% HDD / 13% SSD)
    16 files, 16 threads each, rnd 16k writes, native aio: 27255 write iops (5440% HDD / 49% SSD)
    1 file, 16 threads, rnd 16k reads, native aio: 3794 read iops (2896% HDD / 74% SSD)
    16 files, 16 threads each, rnd 16k reads, native aio: 56839 read iops (11505% HDD / 114% SSD)
    

So it looks like we need to tweak a little our setup, we are not used to such performant hardware ;-)

@baby-gnu

This comment has been minimized.

Copy link
Author

@baby-gnu baby-gnu commented Feb 11, 2020

I found that the bonnie++ error is already solved by #662.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.