Failed to do Latency measurements for jNVMf on HW

[FuchengDum]

Hello,

I have already build jNVMf and run benchmark success on HW. It is important for me to test the Latency and performance . Followed your tips in https://github.com/zrlio/jNVMf/issues/10. The fio command observed:

fio -filename=/dev/ nvme7n1p1 -direct=1 -rw read -m SEQUENTIAL -ioengine=libaio -qd 16 -s 8192 -numjobs=1 -i 3 -n 10 -group_reporting -name=dum

It remind me that engine not loaded, but I really install the libaio1-0.3.109-17.15.x86_64 and libaio-devel-0.3.109-17.15.x86_64 .

Would you tell me whether the fio command is right (if no, please correct me) or any problem about running benchmark.

---

admin@lehi-dirt:~/jNVMf> java -cp target/jnvmf-1.7-jar-with-dependencies.jar:target/jnvmf-1.7-tests.jar -Djnvmf.legacy=true com.ibm.jnvmf.utils.NvmfClientBenchmark -a 192.168.219.8 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2019-08.com.dumnet:test -qd 1 -rw read -s 4096 -qs 64 -H -I
read 4096bytes with QD = 1, time[s] = 3, pattern = RANDOM, runs = 10
log4j:WARN No appenders could be found for logger (com.ibm.disni).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Identify Controller Data:
PCI Vendor ID: 0
PCI Subsystem Vendor ID: 0
Serial Number: 9f61c5e4c474ec19
Model Number: Linux
Controller ID: 2

t[ns] 3000010853, #ops 146204, iops 48734, mean[ns] 20308, tp[MB/s] 190.37, min[ns] 18221, max[ns] 1837523, p1.0% 18928, p10.0% 19134, p20.0% 19226, p30.0% 19278, p40.0% 19315, p50.0% 19351, p75.0% 19509, p90.0% 19762, p95.0% 21468, p98.0% 23864, p99.0% 34480, p99.9% 139207, p99.99% 336263
t[ns] 3000018876, #ops 149431, iops 49810, mean[ns] 19951, tp[MB/s] 194.57, min[ns] 18176, max[ns] 343425, p1.0% 18960, p10.0% 19100, p20.0% 19191, p30.0% 19251, p40.0% 19292, p50.0% 19327, p75.0% 19459, p90.0% 19624, p95.0% 19898, p98.0% 22009, p99.0% 26741, p99.9% 119892, p99.99% 192324
t[ns] 3000012927, #ops 148848, iops 49615, mean[ns] 20033, tp[MB/s] 193.81, min[ns] 18208, max[ns] 882391, p1.0% 18991, p10.0% 19160, p20.0% 19233, p30.0% 19280, p40.0% 19314, p50.0% 19343, p75.0% 19477, p90.0% 19624, p95.0% 19913, p98.0% 22116, p99.0% 27807, p99.9% 119912, p99.99% 191930
t[ns] 3000009064, #ops 148966, iops 49655, mean[ns] 20013, tp[MB/s] 193.96, min[ns] 18193, max[ns] 349483, p1.0% 18964, p10.0% 19118, p20.0% 19209, p30.0% 19265, p40.0% 19303, p50.0% 19335, p75.0% 19463, p90.0% 19610, p95.0% 19860, p98.0% 22077, p99.0% 27729, p99.9% 119893, p99.99% 191591
t[ns] 3000008017, #ops 149035, iops 49678, mean[ns] 20010, tp[MB/s] 194.05, min[ns] 18208, max[ns] 796195, p1.0% 18981, p10.0% 19132, p20.0% 19219, p30.0% 19272, p40.0% 19308, p50.0% 19340, p75.0% 19469, p90.0% 19615, p95.0% 19888, p98.0% 21939, p99.0% 27598, p99.9% 119802, p99.99% 192175
t[ns] 3000002473, #ops 148447, iops 49482, mean[ns] 20086, tp[MB/s] 193.29, min[ns] 18113, max[ns] 2266093, p1.0% 18979, p10.0% 19147, p20.0% 19232, p30.0% 19281, p40.0% 19317, p50.0% 19349, p75.0% 19499, p90.0% 19660, p95.0% 20357, p98.0% 22456, p99.0% 28993, p99.9% 119761, p99.99% 191566
t[ns] 3000004605, #ops 149284, iops 49761, mean[ns] 19963, tp[MB/s] 194.38, min[ns] 18193, max[ns] 811031, p1.0% 18944, p10.0% 19083, p20.0% 19179, p30.0% 19239, p40.0% 19280, p50.0% 19315, p75.0% 19443, p90.0% 19602, p95.0% 20139, p98.0% 22321, p99.0% 27661, p99.9% 119587, p99.99% 191189
t[ns] 3000017266, #ops 149155, iops 49718, mean[ns] 19992, tp[MB/s] 194.21, min[ns] 18180, max[ns] 791290, p1.0% 18971, p10.0% 19125, p20.0% 19214, p30.0% 19269, p40.0% 19307, p50.0% 19340, p75.0% 19473, p90.0% 19620, p95.0% 19930, p98.0% 22137, p99.0% 27667, p99.9% 119476, p99.99% 190673
t[ns] 3000015082, #ops 149280, iops 49759, mean[ns] 19979, tp[MB/s] 194.37, min[ns] 18069, max[ns] 417457, p1.0% 18975, p10.0% 19121, p20.0% 19212, p30.0% 19267, p40.0% 19305, p50.0% 19339, p75.0% 19472, p90.0% 19618, p95.0% 19920, p98.0% 21942, p99.0% 27511, p99.9% 119657, p99.99% 190424
t[ns] 3000017630, #ops 149112, iops 49703, mean[ns] 19999, tp[MB/s] 194.15, min[ns] 18214, max[ns] 801005, p1.0% 18978, p10.0% 19120, p20.0% 19210, p30.0% 19264, p40.0% 19301, p50.0% 19335, p75.0% 19469, p90.0% 19618, p95.0% 19974, p98.0% 22322, p99.0% 27803, p99.9% 119764, p99.99% 190266

Thanks.

[FuchengDum]

Hello,

I am confused about testing the performance of the jNVMf project . Would you tell me how to do the Latency measurement. I want to compare with the FIO test results.

Thank you so much!

[PepperJo]

Sorry for not getting back earlier. I was traveling. A few things: jNVMf is always async, i.e. similar to libaio, also it is always direct. The rest should be similar to fio, you have transfer size, queue depth, sequential/random. Let me know what exactly you don't understand.

[FuchengDum]

Hello, I am appreciate to your help. it is a really new task for me. Following your advice I use the next command to test the Latency of jNVMf.:

admin@lehi-dirt:~/jNVMf> sudo fio -filename=/dev/nvme7n1p1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=async -qd 16 -i 3 -n 10 -group_reporting -name=mytest

However, it can not get the test result. Does there have some problem about the command. if the FIO tools was misused, please correct me what should I do.

And I would like to compare jNVMf with the fio result. The fio tools was used to get the results by follow command:

admin@lehi-dirt:~> fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=200G -numjobs=1 -runtime=30 -group_reporting -name=mytest

Thank you.

[PepperJo]

I'm a bit puzzled by your first fio command. Why do you try to use jNVMf arguments for fio? Neither qd, i or n do exist for fio. Also the correct ioengine would be libaio. I hope you do understand that you cannot run fio to measure performance with jNVMf. jNVMf does not mount a block device.
Also the second fio command is not a sensible latency test. For latency you should use randread with fio. If you want raw device latency measure with 4k or 512byte depending on how your device is formatted.

[FuchengDum]

Thanks to your explanation. I misunderstand the fio tools and use it in a wrong way. Now, my problem may be simple. I am not familiar with how to measure performance with jNVMf.
Because i could not use fio, which tools should i use to test Latency. If it is possible, could your provide me the example or specific command to test jNVMf.
After getting the Latency result, I will do the raw decvice latency measure.

Thank you！

[PepperJo]

You provided the example yourself above with the NvmfClientBenchmark. The arguments I explained in the other issue you referenced. Otherwise there is also a help with -h.

[FuchengDum]

Thanks to your help. The Latency test on HW already finished by using fio and jNVMf.
But I am still a little confused about what you mentioned in the issue #10.
Your tip:

1. Latency and IOPS measurements: use incapsule but not inline
2. Throughput: use inline but not incapsule
3. large IO (>4KB) => inline
4. Small IO (<=4KB) => incapsule
   As you mentioned use incapsule when do latency measurement, May I use the inline data to do the latency measure when IO size is large(>4KB)? And I would like to test the latency when bs=10k, 32k,64k next. what I need to take care?

[PepperJo]

If you do latency measurement for IO sizes >4KB, you should use inline but not incapsule (basically apply tip 3 and 4). Never use both at once (it will not work). IO size always needs to be multiple of formatted sector size on the device, typically 512byte or 4KB, so 10K bs might not work depending on how the device is formatted.

[FuchengDum]

Thank you so much. I will try it right now.