Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tuning & Performance #1067

Closed
aioue opened this issue Feb 24, 2020 · 7 comments
Closed

Tuning & Performance #1067

aioue opened this issue Feb 24, 2020 · 7 comments
Labels
wontfix

Comments

@aioue
Copy link
Contributor

aioue commented Feb 24, 2020

Tested Longhorn 0.7.0 on GKE using fio in directIO mode using dbench.

The bandwidth compared to bare metal/hostpath is somewhat reduced, and IOPS is considerably reduced (see graphs).

Q1: Is this the currently expected performance?
Q2: Is there anything I can do to tune Longhorn and increase IOPS/bandwidth?

image

image

Related #218
@yasker
Copy link
Member

yasker commented Feb 24, 2020

Thanks @aioue for benchmarking. The bandwidth result is not as good as we expected. IOPS result might be caused by something else since it's way too low.

A few questions:

  1. What's the CPU utilization during the test?
  2. Which instance type is used in the test? IOPS was exceptionally high in the bare metal case.
  3. What's the parameter used on starting the test?

@aioue
Copy link
Contributor Author

aioue commented Feb 25, 2020

Each test was run 3 times and the results are averages

Test output

Bare metal output, including CPU info (**click to expand**) 
Working dir: /mnt/disks/ssd0

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
read_iops: Laying out IO file (1 file / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=301021: Mon Feb 17 11:17:07 2020
   read: IOPS=124k, BW=483MiB/s (506MB/s)(7242MiB/15001msec)
   bw (  KiB/s): min=390309, max=550882, per=100.00%, avg=496702.60, stdev=36785.22, samples=30
   iops        : min=97577, max=137720, avg=124175.33, stdev=9196.31, samples=30
  cpu          : usr=23.02%, sys=68.19%, ctx=40306, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1853958,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=483MiB/s (506MB/s), 483MiB/s-483MiB/s (506MB/s-506MB/s), io=7242MiB (7594MB), run=15001-15001msec

Disk stats (read/write):
  sdb: ios=2111685/2, merge=0/1, ticks=739824/8, in_queue=726152, util=99.10%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=301111: Mon Feb 17 11:17:25 2020
  write: IOPS=77.0k, BW=305MiB/s (319MB/s)(4571MiB/15002msec)
   bw (  KiB/s): min=262340, max=328640, per=100.00%, avg=313398.80, stdev=14694.05, samples=30
   iops        : min=65585, max=82160, avg=78349.47, stdev=3673.53, samples=30
  cpu          : usr=14.20%, sys=48.76%, ctx=98280, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,1170002,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=305MiB/s (319MB/s), 305MiB/s-305MiB/s (319MB/s-319MB/s), io=4571MiB (4793MB), run=15002-15002msec

Disk stats (read/write):
  sdb: ios=0/1321328, merge=0/19, ticks=0/956704, in_queue=943124, util=99.49%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=301207: Mon Feb 17 11:17:43 2020
   read: IOPS=5589, BW=699MiB/s (733MB/s)(10.3GiB/15024msec)
   bw (  KiB/s): min=704257, max=728080, per=100.00%, avg=719014.13, stdev=5342.05, samples=30
   iops        : min= 5502, max= 5688, avg=5616.97, stdev=41.62, samples=30
  cpu          : usr=2.08%, sys=11.89%, ctx=66123, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=83983,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=699MiB/s (733MB/s), 699MiB/s-699MiB/s (733MB/s-733MB/s), io=10.3GiB (11.0GB), run=15024-15024msec

Disk stats (read/write):
  sdb: ios=95083/2, merge=0/1, ticks=1081896/12, in_queue=1071880, util=99.84%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=301234: Mon Feb 17 11:18:00 2020
  write: IOPS=3076, BW=385MiB/s (404MB/s)(5790MiB/15034msec)
   bw (  KiB/s): min=335775, max=417098, per=100.00%, avg=396116.57, stdev=18965.13, samples=30
   iops        : min= 2623, max= 3258, avg=3094.20, stdev=148.13, samples=30
  cpu          : usr=5.04%, sys=5.91%, ctx=16537, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,46255,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=385MiB/s (404MB/s), 385MiB/s-385MiB/s (404MB/s-404MB/s), io=5790MiB (6071MB), run=15034-15034msec

Disk stats (read/write):
  sdb: ios=0/52469, merge=0/3, ticks=0/1078940, in_queue=1068304, util=99.63%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.1
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=301311: Mon Feb 17 11:18:18 2020
   read: IOPS=24.6k, BW=95.9MiB/s (101MB/s)(1439MiB/15001msec)
    slat (usec): min=2, max=6528, avg= 7.96, stdev=22.22
    clat (usec): min=26, max=16485, avg=153.88, stdev=198.06
     lat (usec): min=103, max=16490, avg=161.95, stdev=199.29
    clat percentiles (usec):
     |  1.00th=[  111],  5.00th=[  118], 10.00th=[  121], 20.00th=[  126],
     | 30.00th=[  130], 40.00th=[  135], 50.00th=[  137], 60.00th=[  141],
     | 70.00th=[  147], 80.00th=[  155], 90.00th=[  178], 95.00th=[  206],
     | 99.00th=[  273], 99.50th=[  416], 99.90th=[ 4228], 99.95th=[ 4948],
     | 99.99th=[ 6652]
   bw (  KiB/s): min=89569, max=102784, per=100.00%, avg=98659.20, stdev=3035.56, samples=30
   iops        : min=22392, max=25696, avg=24664.50, stdev=758.93, samples=30
  lat (usec)   : 50=0.01%, 100=0.09%, 250=98.14%, 500=1.35%, 750=0.10%
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.09%, 4=0.07%, 10=0.11%, 20=0.01%
  cpu          : usr=6.22%, sys=25.18%, ctx=236458, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=113.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=368442,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=95.9MiB/s (101MB/s), 95.9MiB/s-95.9MiB/s (101MB/s-101MB/s), io=1439MiB (1509MB), run=15001-15001msec

Disk stats (read/write):
  sdb: ios=416146/2, merge=0/1, ticks=59696/4, in_queue=45252, util=96.25%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.1
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=301440: Mon Feb 17 11:18:35 2020
  write: IOPS=25.7k, BW=100MiB/s (105MB/s)(1506MiB/15001msec)
    slat (usec): min=3, max=7120, avg= 7.65, stdev=19.06
    clat (usec): min=2, max=10017, avg=147.07, stdev=262.66
     lat (usec): min=51, max=10033, avg=154.82, stdev=263.41
    clat percentiles (usec):
     |  1.00th=[   57],  5.00th=[   64], 10.00th=[   68], 20.00th=[   73],
     | 30.00th=[   77], 40.00th=[   80], 50.00th=[   83], 60.00th=[   87],
     | 70.00th=[   97], 80.00th=[  129], 90.00th=[  188], 95.00th=[  363],
     | 99.00th=[ 1565], 99.50th=[ 1729], 99.90th=[ 2024], 99.95th=[ 2802],
     | 99.99th=[ 5932]
   bw (  KiB/s): min=96993, max=110530, per=100.00%, avg=103190.57, stdev=3282.34, samples=30
   iops        : min=24248, max=27632, avg=25797.37, stdev=820.50, samples=30
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.34%, 100=71.50%, 250=21.87%
  lat (usec)   : 500=2.21%, 750=0.97%, 1000=0.40%
  lat (msec)   : 2=2.60%, 4=0.08%, 10=0.03%, 20=0.01%
  cpu          : usr=6.24%, sys=24.36%, ctx=165657, majf=0, minf=3
  IO depths    : 1=0.1%, 2=0.1%, 4=112.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,385503,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=1506MiB (1579MB), run=15001-15001msec

Disk stats (read/write):
  sdb: ios=0/434895, merge=0/3, ticks=0/56644, in_queue=42992, util=92.30%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 4 threads
read_seq: Laying out IO file (1 file / 3548MiB)

read_seq: (groupid=0, jobs=1): err= 0: pid=301545: Mon Feb 17 11:19:03 2020
   read: IOPS=175, BW=176MiB/s (185MB/s)(2695MiB/15290msec)
   bw (  KiB/s): min=92716, max=218398, per=25.31%, avg=179588.13, stdev=23607.47, samples=30
   iops        : min=   90, max=  213, avg=174.97, stdev=23.20, samples=30
  cpu          : usr=0.10%, sys=1.24%, ctx=4693, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=111.6%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2680,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=301546: Mon Feb 17 11:19:03 2020
   read: IOPS=175, BW=176MiB/s (185MB/s)(2674MiB/15169msec)
   bw (  KiB/s): min=89133, max=228478, per=25.79%, avg=182982.90, stdev=25074.03, samples=30
   iops        : min=   87, max=  223, avg=178.27, stdev=24.38, samples=30
  cpu          : usr=0.16%, sys=1.19%, ctx=4582, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=112.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2659,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=301547: Mon Feb 17 11:19:03 2020
   read: IOPS=166, BW=168MiB/s (176MB/s)(2543MiB/15163msec)
   bw (  KiB/s): min=110592, max=202752, per=24.25%, avg=172055.50, stdev=19427.19, samples=30
   iops        : min=  108, max=  198, avg=168.00, stdev=18.97, samples=30
  cpu          : usr=0.05%, sys=1.24%, ctx=4291, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=113.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2528,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=301548: Mon Feb 17 11:19:03 2020
   read: IOPS=174, BW=176MiB/s (184MB/s)(2684MiB/15270msec)
   bw (  KiB/s): min=84645, max=241664, per=25.24%, avg=179105.77, stdev=26586.33, samples=30
   iops        : min=   82, max=  236, avg=174.53, stdev=26.02, samples=30
  cpu          : usr=0.05%, sys=1.32%, ctx=4633, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=112.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=2669,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=693MiB/s (727MB/s), 168MiB/s-176MiB/s (176MB/s-185MB/s), io=10.3GiB (11.1GB), run=15163-15290msec

Disk stats (read/write):
  sdb: ios=94996/2, merge=0/1, ticks=8522636/232, in_queue=3824604, util=98.98%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-3.1
Starting 4 threads

write_seq: (groupid=0, jobs=1): err= 0: pid=301652: Mon Feb 17 11:19:21 2020
  write: IOPS=96, BW=97.6MiB/s (102MB/s)(1524MiB/15610msec)
   bw (  KiB/s): min=90112, max=108544, per=24.91%, avg=99691.35, stdev=4560.72, samples=31
   iops        : min=   88, max=  106, avg=97.35, stdev= 4.45, samples=31
  cpu          : usr=0.95%, sys=0.72%, ctx=2661, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.5%, 16=113.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1509,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=301653: Mon Feb 17 11:19:21 2020
  write: IOPS=96, BW=97.6MiB/s (102MB/s)(1528MiB/15656msec)
   bw (  KiB/s): min=92160, max=106496, per=24.80%, avg=99228.90, stdev=4158.29, samples=31
   iops        : min=   90, max=  104, avg=96.90, stdev= 4.06, samples=31
  cpu          : usr=0.86%, sys=0.79%, ctx=2608, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.5%, 16=113.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1513,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=301654: Mon Feb 17 11:19:21 2020
  write: IOPS=97, BW=98.7MiB/s (103MB/s)(1528MiB/15484msec)
   bw (  KiB/s): min=94396, max=113092, per=25.35%, avg=101441.50, stdev=5111.68, samples=30
   iops        : min=   92, max=  110, avg=98.63, stdev= 4.94, samples=30
  cpu          : usr=0.88%, sys=0.81%, ctx=2720, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.5%, 16=113.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1513,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=301655: Mon Feb 17 11:19:21 2020
  write: IOPS=97, BW=98.4MiB/s (103MB/s)(1538MiB/15633msec)
   bw (  KiB/s): min=92160, max=108544, per=25.08%, avg=100345.52, stdev=3885.97, samples=31
   iops        : min=   90, max=  106, avg=97.97, stdev= 3.80, samples=31
  cpu          : usr=1.13%, sys=0.54%, ctx=2656, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.3%, 8=0.5%, 16=112.3%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,1523,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=391MiB/s (410MB/s), 97.6MiB/s-98.7MiB/s (102MB/s-103MB/s), io=6118MiB (6415MB), run=15484-15656msec

Disk stats (read/write):
  sdb: ios=0/54849, merge=0/3, ticks=0/8626748, in_queue=7433876, util=100.00%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=301756: Mon Feb 17 11:19:39 2020
   read: IOPS=77.4k, BW=302MiB/s (317MB/s)(4539MiB/15005msec)
   bw (  KiB/s): min=291078, max=329008, per=100.00%, avg=311404.70, stdev=11471.97, samples=30
   iops        : min=72769, max=82252, avg=77850.90, stdev=2868.09, samples=30
  write: IOPS=25.8k, BW=101MiB/s (106MB/s)(1511MiB/15005msec)
   bw (  KiB/s): min=96129, max=110549, per=100.00%, avg=103676.30, stdev=4372.80, samples=30
   iops        : min=24032, max=27637, avg=25918.77, stdev=1093.20, samples=30
  cpu          : usr=17.68%, sys=59.74%, ctx=131832, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1161871,386829,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=302MiB/s (317MB/s), 302MiB/s-302MiB/s (317MB/s-317MB/s), io=4539MiB (4759MB), run=15005-15005msec
  WRITE: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=1511MiB (1584MB), run=15005-15005msec

Disk stats (read/write):
  sdb: ios=1312144/436793, merge=0/4, ticks=793692/78968, in_queue=857536, util=99.44%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 124k/77.0k. BW: 699MiB/s / 385MiB/s
Average Latency (usec) Read/Write: 161.95/154.82
Sequential Read/Write: 693MiB/s / 391MiB/s
Mixed Random Read/Write IOPS: 77.4k/25.8k
Single node, 1 replica output, including CPU info (**click to expand**) 
Working dir: /data

Testing Read IOPS...
read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process
read_iops: Laying out IO file(s) (1 file(s) / 2048MiB)

read_iops: (groupid=0, jobs=1): err= 0: pid=11: Mon Feb 17 14:59:11 2020
   read: IOPS=2967, BW=11.7MiB/s (12.2MB/s)(174MiB/15017msec)
  cpu          : usr=1.80%, sys=6.69%, ctx=61148, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=113.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=44567,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=11.7MiB/s (12.2MB/s), 11.7MiB/s-11.7MiB/s (12.2MB/s-12.2MB/s), io=174MiB (183MB), run=15017-15017msec

Disk stats (read/write):
  sdc: ios=50592/24, merge=1/3, ticks=1082016/1420, in_queue=529044, util=99.44%


Testing Write IOPS...
write_iops: (g=0): rw=randwrite, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

write_iops: (groupid=0, jobs=1): err= 0: pid=21: Mon Feb 17 14:59:29 2020
  write: IOPS=1801, BW=7223KiB/s (7396kB/s)(107MiB/15106msec)
  cpu          : usr=1.69%, sys=4.11%, ctx=31637, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=109.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,27214,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=7223KiB/s (7396kB/s), 7223KiB/s-7223KiB/s (7396kB/s-7396kB/s), io=107MiB (112MB), run=15106-15106msec

Disk stats (read/write):
  sdc: ios=0/29759, merge=0/8, ticks=0/1083616, in_queue=528764, util=99.46%


Testing Read Bandwidth...
read_bw: (g=0): rw=randread, bs=128KiB-128KiB,128KiB-128KiB,128KiB-128KiB, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

read_bw: (groupid=0, jobs=1): err= 0: pid=31: Mon Feb 17 14:59:47 2020
   read: IOPS=1231, BW=154MiB/s (162MB/s)(2326MiB/15062msec)
  cpu          : usr=0.86%, sys=4.02%, ctx=19589, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=114.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=18546,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=2326MiB (2439MB), run=15062-15062msec

Disk stats (read/write):
  sdc: ios=20812/28, merge=394/4, ticks=1067308/1944, in_queue=528052, util=99.44%


Testing Write Bandwidth...
write_bw: (g=0): rw=randwrite, bs=128KiB-128KiB,128KiB-128KiB,128KiB-128KiB, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

write_bw: (groupid=0, jobs=1): err= 0: pid=41: Mon Feb 17 15:00:04 2020
  write: IOPS=674, BW=84.9MiB/s (88.1MB/s)(1280MiB/15098msec)
  cpu          : usr=2.34%, sys=2.17%, ctx=11392, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=111.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,10180,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=84.9MiB/s (88.1MB/s), 84.9MiB/s-84.9MiB/s (88.1MB/s-88.1MB/s), io=1280MiB (1343MB), run=15098-15098msec

Disk stats (read/write):
  sdc: ios=0/11290, merge=0/49, ticks=0/1075340, in_queue=528528, util=99.47%


Testing Read Latency...
read_latency: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=4
fio-2.17-45-g06cb
Starting 1 process

read_latency: (groupid=0, jobs=1): err= 0: pid=51: Mon Feb 17 15:00:22 2020
   read: IOPS=2145, BW=8581KiB/s (8787kB/s)(126MiB/15003msec)
    slat (usec): min=4, max=9337, avg=29.09, stdev=84.81
    clat (usec): min=584, max=112827, avg=1832.26, stdev=3016.53
     lat (usec): min=616, max=112860, avg=1861.31, stdev=3023.82
    clat percentiles (usec):
     |  1.00th=[  804],  5.00th=[  916], 10.00th=[  980], 20.00th=[ 1080],
     | 30.00th=[ 1144], 40.00th=[ 1224], 50.00th=[ 1304], 60.00th=[ 1400],
     | 70.00th=[ 1512], 80.00th=[ 1720], 90.00th=[ 2448], 95.00th=[ 4384],
     | 99.00th=[10688], 99.50th=[14784], 99.90th=[40192], 99.95th=[68096],
     | 99.99th=[110080]
    lat (usec) : 750=0.41%, 1000=11.40%
    lat (msec) : 2=74.10%, 4=8.44%, 10=4.50%, 20=0.87%, 50=0.23%
    lat (msec) : 100=0.05%, 250=0.02%
  cpu          : usr=2.04%, sys=6.47%, ctx=35444, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=116.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=32182,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=8581KiB/s (8787kB/s), 8581KiB/s-8581KiB/s (8787kB/s-8787kB/s), io=126MiB (132MB), run=15003-15003msec

Disk stats (read/write):
  sdc: ios=37347/28, merge=0/4, ticks=66476/380, in_queue=51188, util=99.09%


Testing Write Latency...
write_latency: (g=0): rw=randwrite, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=4
fio-2.17-45-g06cb
Starting 1 process

write_latency: (groupid=0, jobs=1): err= 0: pid=61: Mon Feb 17 15:00:39 2020
  write: IOPS=1440, BW=5764KiB/s (5902kB/s)(84.5MiB/15007msec)
    slat (usec): min=6, max=15307, avg=35.21, stdev=139.72
    clat (usec): min=742, max=97284, avg=2736.91, stdev=3316.19
     lat (usec): min=778, max=97308, avg=2771.67, stdev=3322.87
    clat percentiles (usec):
     |  1.00th=[  948],  5.00th=[ 1080], 10.00th=[ 1160], 20.00th=[ 1272],
     | 30.00th=[ 1352], 40.00th=[ 1448], 50.00th=[ 1560], 60.00th=[ 1720],
     | 70.00th=[ 2024], 80.00th=[ 3344], 90.00th=[ 6560], 95.00th=[ 8384],
     | 99.00th=[13888], 99.50th=[17792], 99.90th=[39168], 99.95th=[54528],
     | 99.99th=[92672]
    lat (usec) : 750=0.01%, 1000=2.09%
    lat (msec) : 2=67.41%, 4=12.73%, 10=15.51%, 20=1.91%, 50=0.31%
    lat (msec) : 100=0.05%
  cpu          : usr=1.25%, sys=5.13%, ctx=23918, majf=0, minf=3
  IO depths    : 1=0.1%, 2=0.1%, 4=110.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,21622,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=5764KiB/s (5902kB/s), 5764KiB/s-5764KiB/s (5902kB/s-5902kB/s), io=84.5MiB (88.6MB), run=15007-15007msec

Disk stats (read/write):
  sdc: ios=0/23997, merge=0/8, ticks=0/67404, in_queue=51552, util=99.24%


Testing Read Sequential Speed...
read_seq: (g=0): rw=read, bs=1024KiB-1024KiB,1024KiB-1024KiB,1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-2.17-45-g06cb
Starting 4 threads
read_seq: Laying out IO file(s) (1 file(s) / 3548MiB)

read_seq: (groupid=0, jobs=1): err= 0: pid=71: Mon Feb 17 15:01:37 2020
   read: IOPS=65, BW=66.1MiB/s (70.2MB/s)(1027MiB/15346msec)
  cpu          : usr=0.03%, sys=0.56%, ctx=1182, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=112.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1012,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=72: Mon Feb 17 15:01:37 2020
   read: IOPS=66, BW=67.6MiB/s (70.8MB/s)(1039MiB/15391msec)
  cpu          : usr=0.02%, sys=0.57%, ctx=1160, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=110.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1024,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=73: Mon Feb 17 15:01:37 2020
   read: IOPS=65, BW=66.5MiB/s (69.7MB/s)(1021MiB/15368msec)
  cpu          : usr=0.11%, sys=0.48%, ctx=1209, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=112.8%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1006,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
read_seq: (groupid=0, jobs=1): err= 0: pid=74: Mon Feb 17 15:01:37 2020
   read: IOPS=70, BW=71.8MiB/s (75.2MB/s)(1103MiB/15380msec)
  cpu          : usr=0.08%, sys=0.55%, ctx=1266, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.7%, 16=110.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1088,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 66.5MiB/s-71.8MiB/s (69.7MB/s-75.2MB/s), io=4190MiB (4394MB), run=15346-15391msec

Disk stats (read/write):
  sdc: ios=4613/20, merge=0/3, ticks=1085524/6272, in_queue=531332, util=98.47%


Testing Write Sequential Speed...
write_seq: (g=0): rw=write, bs=1024KiB-1024KiB,1024KiB-1024KiB,1024KiB-1024KiB, ioengine=libaio, iodepth=16
...
fio-2.17-45-g06cb
Starting 4 threads

write_seq: (groupid=0, jobs=1): err= 0: pid=84: Mon Feb 17 15:01:56 2020
  write: IOPS=39, BW=40.3MiB/s (42.3MB/s)(648MiB/16100msec)
  cpu          : usr=0.91%, sys=0.25%, ctx=681, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.3%, 4=0.6%, 8=1.3%, 16=108.5%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,633,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=85: Mon Feb 17 15:01:56 2020
  write: IOPS=41, BW=42.4MiB/s (44.4MB/s)(691MiB/16327msec)
  cpu          : usr=0.88%, sys=0.35%, ctx=706, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.3%, 4=0.6%, 8=1.2%, 16=111.1%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,676,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=86: Mon Feb 17 15:01:56 2020
  write: IOPS=39, BW=40.4MiB/s (42.3MB/s)(656MiB/16280msec)
  cpu          : usr=0.87%, sys=0.31%, ctx=681, majf=0, minf=0
  IO depths    : 1=0.2%, 2=0.3%, 4=0.6%, 8=1.2%, 16=107.2%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,641,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
write_seq: (groupid=0, jobs=1): err= 0: pid=87: Mon Feb 17 15:01:56 2020
  write: IOPS=42, BW=43.9MiB/s (45.1MB/s)(721MiB/16465msec)
  cpu          : usr=1.01%, sys=0.26%, ctx=774, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.3%, 4=0.6%, 8=1.1%, 16=106.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=0,706,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=165MiB/s (173MB/s), 40.3MiB/s-43.9MiB/s (42.3MB/s-45.1MB/s), io=2716MiB (2848MB), run=16100-16465msec

Disk stats (read/write):
  sdc: ios=0/2931, merge=0/8, ticks=0/1116752, in_queue=560084, util=99.37%


Testing Read/Write Mixed...
rw_mix: (g=0): rw=randrw, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process

rw_mix: (groupid=0, jobs=1): err= 0: pid=97: Mon Feb 17 15:02:14 2020
   read: IOPS=2081, BW=8341KiB/s (8541kB/s)(123MiB/15102msec)
  write: IOPS=690, BW=2767KiB/s (2833kB/s)(40.9MiB/15102msec)
  cpu          : usr=1.95%, sys=6.58%, ctx=50950, majf=0, minf=1
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=114.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=31439,10434,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=8341KiB/s (8541kB/s), 8341KiB/s-8341KiB/s (8541kB/s-8541kB/s), io=123MiB (129MB), run=15102-15102msec
  WRITE: bw=2767KiB/s (2833kB/s), 2767KiB/s-2767KiB/s (2833kB/s-2833kB/s), io=40.9MiB (42.8MB), run=15102-15102msec

Disk stats (read/write):
  sdc: ios=35982/12020, merge=1/9, ticks=797792/282708, in_queue=529540, util=99.45%


All tests complete.

==================
= Dbench Summary =
==================
Random Read/Write IOPS: 2967/1801. BW: 154MiB/s / 84.9MiB/s
Average Latency (usec) Read/Write: 1861.31/2771.67
Sequential Read/Write: 272MiB/s / 165MiB/s
Mixed Random Read/Write IOPS: 2081/690

VM details

Platform: Google Kubernetes Engine, Ubuntu VMs, 1.14.8-gke.12, europe-west1b
Machine type: custom (2 vCPUs, 5 GB memory), Intel Haswell

Bare metal/hostPath was fast as the VMs had local SSD attached.
image

Config & Test Methodology

3 runs, average of all runs.

Usage: https://github.com/leeliu/dbench#usage

Taken directly from (dbench fio)[https://github.com/leeliu/dbench/blob/master/docker-entrypoint.sh]. Summary:

FIO_SIZE=2G
FIO_DIRECT=1

READ_LATENCY=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --name=read_latency --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=4 --size=$FIO_SIZE --readwrite=randread --time_based --ramp_time=2s --runtime=15s)

WRITE_IOPS=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=write_iops --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=64 --size=$FIO_SIZE --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s)

READ_BW=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=read_bw --filename=$DBENCH_MOUNTPOINT/fiotest --bs=128K --iodepth=64 --size=$FIO_SIZE --readwrite=randread --time_based --ramp_time=2s --runtime=15s)

WRITE_BW=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=write_bw --filename=$DBENCH_MOUNTPOINT/fiotest --bs=128K --iodepth=64 --size=$FIO_SIZE --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s)

@yasker
Copy link
Member

yasker commented Feb 25, 2020

@aioue Thanks for the nice formatted detail data!

I think you're hitting the CPU limitation with only 2 vcpus.

For example, for bare metal, FIO reports CPU for itself as below:

  cpu          : usr=23.02%, sys=68.19%, ctx=40306, majf=0, minf=1

But for Longhorn

  cpu          : usr=1.80%, sys=6.69%, ctx=61148, majf=0, minf=1

Since FIO cannot get more CPU, that means Longhorn is probably using them up and still not enough to catch up with the SSD speed. So Longhorn should benefit from more CPUs.

Also, can you give Ceph a try? IIRC from some users' report, Longhorn's performance should be on-par with Ceph.

@stale
Copy link

stale bot commented Apr 25, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Apr 25, 2020
@yasker yasker added kind/feature Feature request, new feature and removed wontfix kind/feature Feature request, new feature labels Apr 25, 2020
@stale
Copy link

stale bot commented Jun 24, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jun 24, 2020
@stale stale bot closed this as completed Jul 1, 2020
@unixfox
Copy link

unixfox commented Jul 1, 2020 •
edited

Can this issue be reopened? From I can see the community is quite interested to have better performance on longhorn.

@yasker
Copy link
Member

yasker commented Jul 2, 2020

@unixfox I've opened another issue #1541 to track the data path performance improvement of Longhorn. If you've installed ZenHub Chrome Extension, you should be able to see the linked issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aioue @yasker @unixfox