fio --ioengine=libaio --direct=1 --randrepeat=0 --refill_buffers --end_fsync=1 --filename=test-fio.tmp --name=readrandjob --size=12M --bs=4k --rw=randread --iodepth=32 --numjobs=16 --loops=5 readrandjob: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 ... fio-3.1 Starting 16 processes Jobs: 16 (f=16): [r(16)][16.7%][r=59.0MiB/s,w=0KiB/s][r=15.1k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][22.2%][r=61.2MiB/s,w=0KiB/s][r=15.7k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][27.8%][r=55.0MiB/s,w=0KiB/s][r=14.3k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][33.3%][r=58.7MiB/s,w=0KiB/s][r=15.0k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][41.2%][r=62.3MiB/s,w=0KiB/s][r=15.0k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][47.1%][r=54.7MiB/s,w=0KiB/s][r=14.0k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][52.9%][r=56.7MiB/s,w=0KiB/s][r=14.5k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][58.8%][r=62.5MiB/s,w=0KiB/s][r=16.0k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][64.7%][r=55.9MiB/s,w=0KiB/s][r=14.3k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][70.6%][r=56.4MiB/s,w=0KiB/s][r=14.4k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][76.5%][r=59.7MiB/s,w=0KiB/s][r=15.3k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][82.4%][r=59.1MiB/s,w=0KiB/s][r=15.1k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][88.2%][r=56.1MiB/s,w=0KiB/s][r=14.4k,w=0 IOPS][eta 00m:Jobs: 16 (f=16): [r(16)][94.1%][r=57.9MiB/s,w=0KiB/s][r=14.8k,w=0 IOPS][eta 00m:01s] readrandjob: (groupid=0, jobs=1): err= 0: pid=766: Thu Apr 1 20:55:06 2021 read: IOPS=935, BW=3740KiB/s (3830kB/s)(60.0MiB/16426msec) slat (usec): min=6, max=74849, avg=914.85, stdev=5450.13 clat (usec): min=1929, max=121359, avg=33070.68, stdev=15226.90 lat (usec): min=1938, max=149734, avg=33985.98, stdev=16115.67 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 23], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 74], 99.50th=[ 85], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 122] bw ( KiB/s): min= 2730, max= 4370, per=6.20%, avg=3701.06, stdev=329.27, samples=32 iops : min= 682, max= 1092, avg=924.84, stdev=82.33, samples=32 lat (msec) : 2=0.02%, 4=6.50%, 10=10.20%, 20=2.78%, 50=78.18% lat (msec) : 100=1.89%, 250=0.42% cpu : usr=0.44%, sys=1.63%, ctx=3094, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=767: Thu Apr 1 20:55:06 2021 read: IOPS=932, BW=3731KiB/s (3820kB/s)(60.0MiB/16469msec) slat (usec): min=6, max=72117, avg=913.00, stdev=5485.08 clat (usec): min=1711, max=248986, avg=33169.33, stdev=16777.76 lat (usec): min=1721, max=248996, avg=34082.82, stdev=17625.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 17], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 44], 95.00th=[ 46], | 99.00th=[ 80], 99.50th=[ 106], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 2880, max= 4112, per=6.09%, avg=3637.00, stdev=306.21, samples=32 iops : min= 720, max= 1028, avg=909.25, stdev=76.55, samples=32 lat (msec) : 2=0.05%, 4=7.21%, 10=10.75%, 20=2.81%, 50=76.18% lat (msec) : 100=2.38%, 250=0.62% cpu : usr=0.41%, sys=1.75%, ctx=3162, majf=0, minf=54 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=768: Thu Apr 1 20:55:06 2021 read: IOPS=944, BW=3776KiB/s (3867kB/s)(60.0MiB/16270msec) slat (usec): min=6, max=65144, avg=909.80, stdev=5425.92 clat (usec): min=1605, max=122914, avg=32773.35, stdev=14579.10 lat (usec): min=1614, max=149371, avg=33683.63, stdev=15515.01 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 28], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 73], 99.50th=[ 82], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 124] bw ( KiB/s): min= 3104, max= 4464, per=6.29%, avg=3753.66, stdev=326.28, samples=32 iops : min= 776, max= 1116, avg=938.41, stdev=81.58, samples=32 lat (msec) : 2=0.02%, 4=6.61%, 10=10.13%, 20=2.08%, 50=79.81% lat (msec) : 100=1.04%, 250=0.31% cpu : usr=0.25%, sys=1.72%, ctx=2930, majf=0, minf=54 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=769: Thu Apr 1 20:55:06 2021 read: IOPS=932, BW=3731KiB/s (3821kB/s)(60.0MiB/16467msec) slat (usec): min=6, max=72053, avg=921.06, stdev=5481.87 clat (usec): min=1831, max=135496, avg=33206.00, stdev=15895.29 lat (usec): min=1838, max=154908, avg=34127.55, stdev=16704.72 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 23], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 82], 99.50th=[ 96], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 136] bw ( KiB/s): min= 3086, max= 4352, per=6.09%, avg=3632.19, stdev=312.38, samples=32 iops : min= 771, max= 1088, avg=907.81, stdev=78.16, samples=32 lat (msec) : 2=0.05%, 4=6.69%, 10=10.61%, 20=2.16%, 50=77.92% lat (msec) : 100=2.14%, 250=0.44% cpu : usr=0.49%, sys=1.63%, ctx=3043, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=770: Thu Apr 1 20:55:06 2021 read: IOPS=940, BW=3762KiB/s (3852kB/s)(60.0MiB/16332msec) slat (usec): min=6, max=55646, avg=914.54, stdev=5442.75 clat (usec): min=1854, max=133704, avg=33003.27, stdev=14726.40 lat (usec): min=1863, max=133715, avg=33918.58, stdev=15689.28 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 28], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 74], 99.50th=[ 80], 99.90th=[ 94], 99.95th=[ 94], | 99.99th=[ 133] bw ( KiB/s): min= 3032, max= 4440, per=6.27%, avg=3740.88, stdev=316.17, samples=32 iops : min= 758, max= 1110, avg=934.72, stdev=79.16, samples=32 lat (msec) : 2=0.03%, 4=7.18%, 10=10.07%, 20=1.49%, 50=78.61% lat (msec) : 100=2.58%, 250=0.04% cpu : usr=0.61%, sys=1.42%, ctx=3030, majf=0, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=771: Thu Apr 1 20:55:06 2021 read: IOPS=933, BW=3732KiB/s (3822kB/s)(60.0MiB/16461msec) slat (usec): min=6, max=76105, avg=917.39, stdev=5450.17 clat (usec): min=1736, max=128436, avg=33185.39, stdev=15142.72 lat (usec): min=1750, max=145511, avg=34103.39, stdev=16066.69 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 26], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 46], | 99.00th=[ 74], 99.50th=[ 79], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 121] bw ( KiB/s): min= 3006, max= 4200, per=6.13%, avg=3656.88, stdev=303.99, samples=32 iops : min= 751, max= 1050, avg=913.63, stdev=76.05, samples=32 lat (msec) : 2=0.02%, 4=6.81%, 10=10.01%, 20=2.32%, 50=77.49% lat (msec) : 100=3.11%, 250=0.25% cpu : usr=0.53%, sys=1.48%, ctx=2965, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=772: Thu Apr 1 20:55:06 2021 read: IOPS=937, BW=3750KiB/s (3840kB/s)(60.0MiB/16383msec) slat (usec): min=6, max=62174, avg=915.38, stdev=5453.49 clat (usec): min=1916, max=126556, avg=32938.63, stdev=14451.72 lat (usec): min=1925, max=126565, avg=33854.48, stdev=15431.07 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 27], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 71], 99.50th=[ 77], 99.90th=[ 84], 99.95th=[ 88], | 99.99th=[ 122] bw ( KiB/s): min= 3176, max= 4392, per=6.24%, avg=3726.81, stdev=316.14, samples=32 iops : min= 794, max= 1098, avg=931.47, stdev=79.11, samples=32 lat (msec) : 2=0.03%, 4=6.73%, 10=10.10%, 20=1.88%, 50=79.18% lat (msec) : 100=2.05%, 250=0.03% cpu : usr=0.59%, sys=1.39%, ctx=3026, majf=0, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=773: Thu Apr 1 20:55:06 2021 read: IOPS=942, BW=3769KiB/s (3860kB/s)(60.0MiB/16301msec) slat (usec): min=6, max=47886, avg=907.21, stdev=5408.09 clat (usec): min=1818, max=196518, avg=32871.71, stdev=14255.09 lat (usec): min=1863, max=196527, avg=33779.45, stdev=15153.88 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 29], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 66], 99.50th=[ 73], 99.90th=[ 83], 99.95th=[ 89], | 99.99th=[ 126] bw ( KiB/s): min= 3364, max= 4744, per=6.31%, avg=3767.50, stdev=314.97, samples=32 iops : min= 841, max= 1186, avg=941.41, stdev=78.89, samples=32 lat (msec) : 2=0.03%, 4=6.73%, 10=10.35%, 20=1.59%, 50=79.51% lat (msec) : 100=1.74%, 250=0.05% cpu : usr=0.52%, sys=1.60%, ctx=3020, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=774: Thu Apr 1 20:55:06 2021 read: IOPS=943, BW=3773KiB/s (3863kB/s)(60.0MiB/16285msec) slat (usec): min=6, max=67566, avg=909.31, stdev=5426.38 clat (usec): min=1809, max=131299, avg=32857.77, stdev=14399.18 lat (usec): min=1820, max=140926, avg=33767.67, stdev=15328.58 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 27], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 44], 95.00th=[ 45], | 99.00th=[ 59], 99.50th=[ 74], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 3262, max= 4448, per=6.27%, avg=3742.81, stdev=299.31, samples=32 iops : min= 815, max= 1112, avg=935.47, stdev=74.92, samples=32 lat (msec) : 2=0.03%, 4=7.23%, 10=9.90%, 20=1.48%, 50=79.92% lat (msec) : 100=1.25%, 250=0.20% cpu : usr=0.47%, sys=1.65%, ctx=3010, majf=0, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=775: Thu Apr 1 20:55:06 2021 read: IOPS=935, BW=3740KiB/s (3830kB/s)(60.0MiB/16427msec) slat (usec): min=6, max=80500, avg=923.39, stdev=5531.82 clat (msec): min=2, max=113, avg=33.07, stdev=15.37 lat (msec): min=2, max=146, avg=33.99, stdev=16.21 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 24], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 44], | 99.00th=[ 78], 99.50th=[ 103], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 113] bw ( KiB/s): min= 3086, max= 4320, per=6.20%, avg=3701.19, stdev=341.24, samples=32 iops : min= 771, max= 1080, avg=924.81, stdev=85.49, samples=32 lat (msec) : 4=6.99%, 10=9.60%, 20=2.65%, 50=78.83%, 100=1.35% lat (msec) : 250=0.59% cpu : usr=0.34%, sys=1.61%, ctx=2988, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=776: Thu Apr 1 20:55:06 2021 read: IOPS=945, BW=3783KiB/s (3873kB/s)(60.0MiB/16243msec) slat (usec): min=6, max=48461, avg=900.71, stdev=5387.54 clat (usec): min=1938, max=206244, avg=32797.61, stdev=14571.44 lat (usec): min=1958, max=206253, avg=33699.55, stdev=15487.69 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 24], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 68], 99.50th=[ 82], 99.90th=[ 87], 99.95th=[ 87], | 99.99th=[ 120] bw ( KiB/s): min= 3333, max= 4401, per=6.32%, avg=3773.72, stdev=292.93, samples=32 iops : min= 833, max= 1100, avg=942.97, stdev=73.34, samples=32 lat (msec) : 2=0.03%, 4=6.62%, 10=10.59%, 20=1.82%, 50=78.76% lat (msec) : 100=2.16%, 250=0.03% cpu : usr=0.44%, sys=1.55%, ctx=3055, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=777: Thu Apr 1 20:55:06 2021 read: IOPS=937, BW=3749KiB/s (3839kB/s)(60.0MiB/16388msec) slat (usec): min=6, max=58461, avg=921.59, stdev=5457.95 clat (usec): min=1741, max=230255, avg=33030.89, stdev=14937.24 lat (usec): min=1751, max=230269, avg=33953.20, stdev=15895.03 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 26], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 77], 99.50th=[ 81], 99.90th=[ 96], 99.95th=[ 96], | 99.99th=[ 101] bw ( KiB/s): min= 3302, max= 4496, per=6.23%, avg=3718.06, stdev=298.01, samples=32 iops : min= 825, max= 1124, avg=929.16, stdev=74.60, samples=32 lat (msec) : 2=0.04%, 4=6.86%, 10=10.38%, 20=1.33%, 50=78.95% lat (msec) : 100=2.42%, 250=0.01% cpu : usr=0.54%, sys=1.59%, ctx=2985, majf=0, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=778: Thu Apr 1 20:55:06 2021 read: IOPS=940, BW=3762KiB/s (3852kB/s)(60.0MiB/16333msec) slat (usec): min=6, max=71922, avg=914.16, stdev=5470.05 clat (usec): min=1803, max=117628, avg=32938.23, stdev=15507.35 lat (usec): min=1858, max=149203, avg=33852.87, stdev=16405.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 26], | 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 41], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 78], 99.50th=[ 103], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 2829, max= 4448, per=6.27%, avg=3740.97, stdev=368.56, samples=32 iops : min= 707, max= 1112, avg=935.12, stdev=92.18, samples=32 lat (msec) : 2=0.03%, 4=7.07%, 10=10.20%, 20=2.01%, 50=78.52% lat (msec) : 100=1.63%, 250=0.55% cpu : usr=0.44%, sys=1.54%, ctx=3049, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=779: Thu Apr 1 20:55:06 2021 read: IOPS=935, BW=3743KiB/s (3833kB/s)(60.0MiB/16415msec) slat (usec): min=6, max=68524, avg=922.17, stdev=5475.32 clat (usec): min=1969, max=152354, avg=33118.78, stdev=14990.57 lat (usec): min=1978, max=152369, avg=34041.55, stdev=15913.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 27], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 45], | 99.00th=[ 74], 99.50th=[ 84], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 130] bw ( KiB/s): min= 3118, max= 4682, per=6.20%, avg=3700.19, stdev=342.80, samples=32 iops : min= 779, max= 1170, avg=924.62, stdev=85.71, samples=32 lat (msec) : 2=0.01%, 4=7.01%, 10=9.54%, 20=2.12%, 50=78.79% lat (msec) : 100=2.28%, 250=0.25% cpu : usr=0.27%, sys=1.66%, ctx=2893, majf=0, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=780: Thu Apr 1 20:55:06 2021 read: IOPS=935, BW=3743KiB/s (3833kB/s)(60.0MiB/16415msec) slat (usec): min=6, max=54750, avg=915.11, stdev=5451.50 clat (usec): min=1824, max=178910, avg=33007.74, stdev=15317.04 lat (usec): min=1832, max=178923, avg=33923.29, stdev=16266.88 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 24], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 46], | 99.00th=[ 75], 99.50th=[ 82], 99.90th=[ 128], 99.95th=[ 129], | 99.99th=[ 144] bw ( KiB/s): min= 2985, max= 4394, per=6.21%, avg=3706.00, stdev=313.09, samples=32 iops : min= 746, max= 1098, avg=926.16, stdev=78.20, samples=32 lat (msec) : 2=0.01%, 4=6.93%, 10=10.22%, 20=2.54%, 50=77.76% lat (msec) : 100=2.33%, 250=0.20% cpu : usr=0.37%, sys=1.73%, ctx=3053, majf=0, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 readrandjob: (groupid=0, jobs=1): err= 0: pid=781: Thu Apr 1 20:55:06 2021 read: IOPS=936, BW=3747KiB/s (3837kB/s)(60.0MiB/16398msec) slat (usec): min=6, max=68972, avg=920.76, stdev=5463.87 clat (usec): min=1666, max=230778, avg=32982.58, stdev=14616.39 lat (usec): min=1682, max=230793, avg=33903.79, stdev=15513.23 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 28], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 40], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 43], 95.00th=[ 44], | 99.00th=[ 69], 99.50th=[ 80], 99.90th=[ 112], 99.95th=[ 113], | 99.99th=[ 121] bw ( KiB/s): min= 3032, max= 4386, per=6.26%, avg=3733.75, stdev=333.60, samples=32 iops : min= 758, max= 1096, avg=933.16, stdev=83.41, samples=32 lat (msec) : 2=0.06%, 4=6.94%, 10=9.34%, 20=1.86%, 50=80.09% lat (msec) : 100=1.32%, 250=0.39% cpu : usr=0.49%, sys=1.46%, ctx=2912, majf=0, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.5%, 32=99.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=15360,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=58.3MiB/s (61.1MB/s), 3731KiB/s-3783KiB/s (3820kB/s-3873kB/s), io=960MiB (1007MB), run=16243-16469msec Disk stats (read/write): sda: ios=206288/2, merge=35171/2, ticks=2346968/8, in_queue=2355696, util=99.40%