Performance test result on validation is unstable

[classcwnd]

I tested several times for the same big datastore on data validation.
But the performance results are not stable. e.g., I got 48s, 71s, 80s, ...
The first time after yanglint starting up always got the minimum result.
attached the datastore3.zip, it is 50k interfaces.

> add ietf-interfaces@2014-05-08.yin
> 
> add iana-if-type.yin
> add ericsson-contexts-ipos@2015-12-07.yin
> add ericsson-context-ipos@2015-12-07.yin
> add ericsson-interfaces-ext-ipos@2015-12-07.yin
> add ericsson-ethernet-ext-ipos@2015-12-07.yin
> 
> data -s datastore3.xml

[rkrejci]

I got the same results, but I really don't know why. I spent on this two days, did some optimizations and bugfixing, but it is still similar - loading the same datastore 2 times takes more than 2 times of loading time of the same datastore only once.

I've used the attached code to simulate the problem and from the perf stats you can see that the number of instructions are approximately corresponding, but loading the datastore once takes 57,7s while loading it twice takes 130,5s (instead of expected 115s). I've inspected even the perf's detailed records, I've tried to increase the process priority (nice value), but I'don't know where is the problem.

$ sudo perf stat ./repeat 0
 Performance counter stats for './repeat 0':

         14,087435      task-clock (msec)         #    0,981 CPUs utilized          
                 0      context-switches          #    0,000 K/sec                  
                 0      cpu-migrations            #    0,000 K/sec                  
               201      page-faults               #    0,014 M/sec                  
        35 858 939      cycles                    #    2,545 GHz                    
   <not supported>      stalled-cycles-frontend  
   <not supported>      stalled-cycles-backend   
        51 240 113      instructions              #    1,43  insns per cycle        
        10 435 023      branches                  #  740,733 M/sec                  
           124 104      branch-misses             #    1,19% of all branches        

       0,014354240 seconds time elapsed

$ sudo perf stat ./repeat 1
 Performance counter stats for './repeat 1':

      57809,400321      task-clock (msec)         #    1,001 CPUs utilized          
                93      context-switches          #    0,002 K/sec                  
                16      cpu-migrations            #    0,000 K/sec                  
             8 476      page-faults               #    0,147 K/sec                  
   177 354 791 754      cycles                    #    3,068 GHz                    
   <not supported>      stalled-cycles-frontend  
   <not supported>      stalled-cycles-backend   
   167 631 918 696      instructions              #    0,95  insns per cycle        
    37 915 686 371      branches                  #  655,874 M/sec                  
         8 839 854      branch-misses             #    0,02% of all branches        

      57,752154710 seconds time elapsed

$ sudo perf stat ./repeat 2
 Performance counter stats for './repeat 2':

     130676,185359      task-clock (msec)         #    1,001 CPUs utilized          
               192      context-switches          #    0,001 K/sec                  
                10      cpu-migrations            #    0,000 K/sec                  
             9 275      page-faults               #    0,071 K/sec                  
   400 423 193 269      cycles                    #    3,064 GHz                    
   <not supported>      stalled-cycles-frontend  
   <not supported>      stalled-cycles-backend   
   335 334 352 450      instructions              #    0,84  insns per cycle        
    75 843 820 226      branches                  #  580,395 M/sec                  
        17 772 382      branch-misses             #    0,02% of all branches        

     130,546050744 seconds time elapsed

Any ideas?

[rkrejci]

I did some more tests with modified repeat.c - I changed the for loop to be around all libyang functions, so it is creating context, loading schemas, loading datastore and then cleaning up in each loop. And the results are the same, there is still some time overhead. Therefore I believe that the issue is caused outside the libyang functions (maybe memory allocation?) cused by long running process instead of repeating data parsing.

BTW, I saw also some time differences even in case of running the test with the same parameters. According to the perf output, it seems that the differences were caused by CPU frequency which were changing (in my case) from 2.5 to 3.1 GHz.

[classcwnd]

Sorry for late, I just had time to look into libyang again. I think your investigation results made sense. Maybe it is caused by CPU frequency changing.
I think we can add some profiling code into libyang code to check which parts are time consuming.
We have improved libnetconf performance a lot in edit_config and xml_diff by adding profiling code.

BTW, I didn't see similar functionality of edit_config and xml_diff realized by APIs of libyang. Where are there parts, will they be in libnetconf2?

[rkrejci]

libyang is supposed to provide support for its use in NETCONF tools, but it is being separated from the NETCONF. Therefore, libyang allows you to manipulate with the data tree (add/remove/change a node), but implementation of NETCONF's edit-config is out of libyang scope.

libnetconf2, in contrast to libnetconf, does not cover datastore implementation. We are going to implement edit-config and other NETCONF operations in Netopeer2 using sysrepo as a datastore.