Question about using lkp ncompare

[schrht]

I'm so sorry to disturb you. But after studying many articles (blog/pdf/slides/README.md) I still can not get lkp ncompare to run successfully.

For example: in https://01.org/zh/node/15555 it says:
# lkp ncompare -s commit=9e66317d3c92ddaab330c125dfe9d06eee268aff -o commit=a11c3148ba6b8b4cac4876b7269d570260d0bd41
But I have no idea about where to get the commit id.

Does lkp ncompare command support comparing other things beside the kernel? for example, compare 2 versions of a driver or compare 2 different NVMe devices?
Such as: After testing fio-basic against 2 devices, is there any available tools to tell the performance difference?

[root@ip-10-22-2-226 lkp]# lkp result fio-basic
/lkp/result/fio-basic/10s-8t-1SSD-xfs-read-4k-sync-5g/ip-10-22-2-226.us-west-2.compute.internal/rhel/x86_64-rhel-7.2/gcc-7/3.10.0-862.el7.x86_64/0/
/lkp/result/fio-basic/10s-8t-1SSD-xfs-read-4k-sync-5g/ip-10-22-2-226.us-west-2.compute.internal/rhel/x86_64-rhel-7.2/gcc-7/3.10.0-862.el7.x86_64/1/

Thank you in advance!

[hying-caritas]

Commit is the kernel version you use, for example "3.10.0-862.el7.x86_64" in your results. But you need to compare different kernel versions, so you need to run the test in another kernel version too.

It's possible to compare with other axis key. Please try

$ lkp ncompare -h

And use "-c" to specify axis key.

[schrht]

Will try to upgrade the kernel and run a benchmark (unixbench) to compare.
Sorry I didn't get the idea to run lkp ncompare against the following results:

[root@ip-10-22-2-226 lkp]# lkp result fio-basic
/lkp/result/fio-basic/10s-8t-1SSD-xfs-read-4k-sync-5g/ip-10-22-2-226.us-west-2.compute.internal/rhel/x86_64-rhel-7.2/gcc-7/3.10.0-862.el7.x86_64/0/
/lkp/result/fio-basic/10s-8t-1SSD-xfs-read-4k-sync-5g/ip-10-22-2-226.us-west-2.compute.internal/rhel/x86_64-rhel-7.2/gcc-7/3.10.0-862.el7.x86_64/1/

Could you help clarify the usage and/or give me an example?

Sorry for inconvenience.

[schrht]

After trying another kernel, I still can not run lkp ncompare successfully.

[root@ip-10-22-2-226 gcc]# pwd
/lkp/result/unixbench/30s-_8_-__whetstone-double____shell1___/ip-10-22-2-226.us-west-2.compute.internal/rhel/defconfig/gcc
[root@ip-10-22-2-226 gcc]# ls
3.10.0-862.19.1.el7.x86_64  3.10.0-862.el7.x86_64
[root@ip-10-22-2-226 gcc]# 
[root@ip-10-22-2-226 gcc]# cat ./3.10.0-862.19.1.el7.x86_64/0/stats.json 
{
  "unixbench.score": 1913.7,
  "unixbench.workload": 33146292187.8,
  "unixbench.time.user_time": 332.02,
  "unixbench.time.system_time": 455.92,
  "unixbench.time.percent_of_cpu_this_job_got": 155,
  "unixbench.time.elapsed_time": 506.3,
  "unixbench.time.elapsed_time.max": 506.3,
  "unixbench.time.maximum_resident_set_size": 6176,
  "unixbench.time.major_page_faults": 14,
  "unixbench.time.minor_page_faults": 88560475,
  "unixbench.time.voluntary_context_switches": 8687259,
  "unixbench.time.involuntary_context_switches": 77727,
  "unixbench.time.file_system_inputs": 7728,
  "unixbench.time.file_system_outputs": 2453808,
  "unixbench.time.page_size": 4096,
  "time.user_time": 332.02,
  "time.system_time": 455.92,
  "time.percent_of_cpu_this_job_got": 155,
  "time.elapsed_time": 506.3,
  "time.elapsed_time.max": 506.3,
  "time.maximum_resident_set_size": 6176,
  "time.major_page_faults": 14,
  "time.minor_page_faults": 88560475,
  "time.voluntary_context_switches": 8687259,
  "time.involuntary_context_switches": 77727,
  "time.file_system_inputs": 7728,
  "time.file_system_outputs": 2453808,
  "time.page_size": 4096
}[root@ip-10-22-2-226 gcc]# cat ./3.10.0-862.el7.x86_64/0/stats.json 
{
  "unixbench.score": 1854.5,
  "unixbench.workload": 33124226438.600002,
  "unixbench.time.user_time": 328.77,
  "unixbench.time.system_time": 459.53,
  "unixbench.time.percent_of_cpu_this_job_got": 155,
  "unixbench.time.elapsed_time": 506.46,
  "unixbench.time.elapsed_time.max": 506.46,
  "unixbench.time.maximum_resident_set_size": 6184,
  "unixbench.time.minor_page_faults": 86516980,
  "unixbench.time.voluntary_context_switches": 7950482,
  "unixbench.time.involuntary_context_switches": 77330,
  "unixbench.time.file_system_outputs": 2435096,
  "unixbench.time.page_size": 4096,
  "time.user_time": 328.77,
  "time.system_time": 459.53,
  "time.percent_of_cpu_this_job_got": 155,
  "time.elapsed_time": 506.46,
  "time.elapsed_time.max": 506.46,
  "time.maximum_resident_set_size": 6184,
  "time.minor_page_faults": 86516980,
  "time.voluntary_context_switches": 7950482,
  "time.involuntary_context_switches": 77330,
  "time.file_system_outputs": 2435096,
  "time.page_size": 4096
}

The lkp ncompare command has no outputs.

[root@ip-10-22-2-226 gcc]# lkp result unixbench
/lkp/result/unixbench/30s-_8_-__whetstone-double____shell1___/ip-10-22-2-226.us-west-2.compute.internal/rhel/defconfig/gcc/3.10.0-862.19.1.el7.x86_64/0/
/lkp/result/unixbench/30s-_8_-__whetstone-double____shell1___/ip-10-22-2-226.us-west-2.compute.internal/rhel/defconfig/gcc/3.10.0-862.el7.x86_64/0/
[root@ip-10-22-2-226 gcc]# lkp ncompare -s commit='3.10.0-862.el7.x86_64' -o commit='3.10.0-900.el7.x86_64'
[root@ip-10-22-2-226 gcc]# 

[yhuang-intel]

At least 3 runs for 1 kernel are needed to compare.

[schrht]

At least 3 runs for 1 kernel are needed to compare.

OK. Will try it. Suggest to add some outputs there if no enough samples. Thanks!

[schrht]

At least 3 runs for 1 kernel are needed to compare.

OK. Will try it. Suggest to add some outputs there if no enough samples. Thanks!

Still not work after 3 runs, is there any other mistakes?

[root@ip-10-22-2-226 gcc]# pwd
/lkp/result/unixbench/30s-_8_-__whetstone-double____shell1___/ip-10-22-2-226.us-west-2.compute.internal/rhel/defconfig/gcc
[root@ip-10-22-2-226 gcc]# ls -lad */*
drwxrwsr-x. 3 root root 4096 Sep 29 03:53 3.10.0-862.19.1.el7.x86_64/0
drwxrwsr-x. 3 root root 4096 Sep 29 05:38 3.10.0-862.19.1.el7.x86_64/1
drwxrwsr-x. 3 root root 4096 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/2
-rw-r--r--. 1 root root 1188 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/avg.csv
-rw-r--r--. 1 root root 1358 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/avg.json
-rw-r--r--. 1 root root  399 Sep 29 03:53 3.10.0-862.19.1.el7.x86_64/desc.yaml
-rw-r--r--. 1 root root 1816 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/matrix.csv
-rw-r--r--. 1 root root 2522 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/matrix.json
-rw-r--r--. 1 root root 1258 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/stddev.csv
-rw-r--r--. 1 root root 1428 Sep 29 05:46 3.10.0-862.19.1.el7.x86_64/stddev.json
drwxrwsr-x. 3 root root 4096 Sep 29 02:52 3.10.0-862.el7.x86_64/0
drwxrwsr-x. 3 root root 4096 Sep 29 06:04 3.10.0-862.el7.x86_64/1
drwxrwsr-x. 3 root root 4096 Sep 29 06:14 3.10.0-862.el7.x86_64/2
-rw-r--r--. 1 root root 1235 Sep 29 06:14 3.10.0-862.el7.x86_64/avg.csv
-rw-r--r--. 1 root root 1405 Sep 29 06:14 3.10.0-862.el7.x86_64/avg.json
-rw-r--r--. 1 root root  394 Sep 29 02:52 3.10.0-862.el7.x86_64/desc.yaml
-rw-r--r--. 1 root root 1824 Sep 29 06:14 3.10.0-862.el7.x86_64/matrix.csv
-rw-r--r--. 1 root root 2558 Sep 29 06:14 3.10.0-862.el7.x86_64/matrix.json
-rw-r--r--. 1 root root 1257 Sep 29 06:14 3.10.0-862.el7.x86_64/stddev.csv
-rw-r--r--. 1 root root 1427 Sep 29 06:14 3.10.0-862.el7.x86_64/stddev.json
[root@ip-10-22-2-226 gcc]# 
[root@ip-10-22-2-226 gcc]# lkp ncompare -s commit='3.10.0-862.el7.x86_64' -o commit='3.10.0-900.el7.x86_64'
[root@ip-10-22-2-226 gcc]# echo $?
0
[root@ip-10-22-2-226 gcc]# 

[yhuang-intel]

Try

$ lkp ncompare -s commit='3.10.0-862.19.1.el7.x86_64' -o commit='3.10.0-862.el7.x86_64'

and

$ lkp ncompare -s commit='3.10.0-862.19.1.el7.x86_64' -o commit='3.10.0-862.el7.x86_64' -a

?

[schrht]

The command with -a worked.

[root@ip-10-22-2-226 ~]# lkp ncompare -s commit='3.10.0-862.19.1.el7.x86_64' -o commit='3.10.0-862.el7.x86_64'
[root@ip-10-22-2-226 ~]# echo $?
0
[root@ip-10-22-2-226 ~]# lkp ncompare -s commit='3.10.0-862.19.1.el7.x86_64' -o commit='3.10.0-862.el7.x86_64' -a
=========================================================================================
compiler/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc/defconfig/[8]/rhel/30s/ip-10-22-2-226.us-west-2.compute.internal/["whetstone-double", "shell1", "shell8", "pipe", "context1", "spawn", "syscall", "dhry2reg", "fstime", "fsbuffer", "fsdisk"]/unixbench

commit: 
  3.10.0-862.19.1.el7.x86_64
  3.10.0-862.el7.x86_64

3.10.0-862.19.1.      3.10.0-862.el7.x86_64 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1901            -0.9%       1885        unixbench.score
    506.37            +0.0%     506.48        unixbench.time.elapsed_time
    506.37            +0.0%     506.48        unixbench.time.elapsed_time.max
      2576 ±141%      +8.9%       2806 ±141%  unixbench.time.file_system_inputs
   2456501            +0.9%    2477546        unixbench.time.file_system_outputs
     77728            -0.3%      77488        unixbench.time.involuntary_context_switches
      4.67 ±141%     +14.3%       5.33 ±141%  unixbench.time.major_page_faults
      6178            +0.0%       6181        unixbench.time.maximum_resident_set_size
  88371177            -0.1%   88292389        unixbench.time.minor_page_faults
      4096            +0.0%       4096        unixbench.time.page_size
    155.00            +0.0%     155.00        unixbench.time.percent_of_cpu_this_job_got
    456.57            +0.0%     456.79        unixbench.time.system_time
    331.53            -0.0%     331.45        unixbench.time.user_time
   8632176            -2.6%    8407320 ±  3%  unixbench.time.voluntary_context_switches
 3.311e+10            -0.0%   3.31e+10        unixbench.workload
    506.37            +0.0%     506.48        time.elapsed_time
    506.37            +0.0%     506.48        time.elapsed_time.max
      2576 ±141%      +8.9%       2806 ±141%  time.file_system_inputs
   2456501            +0.9%    2477546        time.file_system_outputs
     77728            -0.3%      77488        time.involuntary_context_switches
      4.67 ±141%     +14.3%       5.33 ±141%  time.major_page_faults
      6178            +0.0%       6181        time.maximum_resident_set_size
  88371177            -0.1%   88292389        time.minor_page_faults
      4096            +0.0%       4096        time.page_size
    155.00            +0.0%     155.00        time.percent_of_cpu_this_job_got
    456.57            +0.0%     456.79        time.system_time
    331.53            -0.0%     331.45        time.user_time
   8632176            -2.6%    8407320 ±  3%  time.voluntary_context_switches

[root@ip-10-22-2-226 ~]# 

[yhuang-intel]

I guess the two kernel versions are too similar. There's no performance difference, so you need "-a".

[schrht]

May I run lkp-tests on 2 different hosts (the same capacity) and use lkp ncompare to summary the benchmark results? If so, I can compare RHEL7.5 against RHEL7.6, there should be some difference.

[yhuang-intel]

It's not easy to merge results from different machine now yet. That will be a TODO for us. Maybe you can close this one and submit a new issue for that? So we will keep track of it and make it resolved.

[schrht]

It's not easy to merge results from different machine now yet. That will be a TODO for us. Maybe you can close this one and submit a new issue for that? So we will keep track of it and make it resolved.

I'm all for it, this feature would be very useful.

BTW, still one more question:
Can lkp ncompare be used for comparing the KPI of 2 different SSD devices based on the same kernel? I guess no, but if it can, please let me know. Thanks!

[hying-caritas]

It's possible, but tricky. Firstly you need a way to differentiate 2 SSD devices. Now in host file, you cannot do that. That is a feature request. It's not only for lkp ncompare, but also for lkp run. So maybe create another issue for this too. Thanks for pointing this out.

[schrht]

It's possible, but tricky. Firstly you need a way to differentiate 2 SSD devices. Now in host file, you cannot do that. That is a feature request. It's not only for lkp ncompare, but also for lkp run. So maybe create another issue for this too. Thanks for pointing this out.

Thanks, will do!