Varint cython

[maxnoe]

No description provided.

[maxnoe]

I profiled it, for reading every object in the sst1m simtel file, we get a speedup of ~25.

Before

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    160/1    0.001    0.000   96.834   96.834 {built-in method builtins.exec}
        1    0.019    0.019   96.834   96.834 time_it.py:1(<module>)
      129    0.001    0.000   93.437    0.724 objects.py:712(parse_data_field)
      129    0.261    0.002   93.431    0.724 objects.py:759(_parse_in_not_zero_suppressed_mode)
   167184    7.541    0.000   93.169    0.001 tools.py:108(read_vector_of_uint32_scount_differential_optimized)
26188829/8898340   15.231    0.000   87.458    0.000 base.py:181(read)
52188919/26256293   15.993    0.000   62.605    0.000 base.py:239(tell)
 26257316    4.796    0.000   46.614    0.000 base.py:63(tell)
 26258339    7.614    0.000   41.820    0.000 {method 'tell' of '_io._IOBase' objects}
 26285823   11.027    0.000   34.247    0.000 gzip.py:352(seek)
 26285823    8.615    0.000   16.199    0.000 {method 'seek' of '_io.BufferedReader' objects}
  8900387    2.346    0.000    9.779    0.000 base.py:66(read)
 35186211    6.297    0.000    9.383    0.000 _compression.py:12(_check_not_closed)
 26285823    5.943    0.000    7.579    0.000 _compression.py:63(seekable)
  8900388    3.597    0.000    7.432    0.000 gzip.py:271(read)
 35186213    3.086    0.000    3.086    0.000 gzip.py:298(closed)
 26285823    1.636    0.000    1.636    0.000 gzip.py:107(seekable)
   214620    0.129    0.000    1.534    0.000 tools.py:19(read_from)
8902382/8900467    1.367    0.000    1.474    0.000 {method 'read' of '_io.BufferedReader' objects}
      129    0.002    0.000    1.192    0.009 objects.py:903(parse_data_field)
      129    0.043    0.000    1.169    0.009 objects.py:940(_parse_list_type_1)
       70    0.067    0.001    1.159    0.017 objects.py:242(parse_data_field)
    96114    0.070    0.000    0.550    0.000 tools.py:5(read_array)
44956/15116    0.023    0.000    0.528    0.000 search_utils.py:42(yield_all_objects_depth_first)
    14359    0.031    0.000    0.475    0.000 base.py:208(__next__)
    17654    0.009    0.000    0.459    0.000 tools.py:50(read_utf8_like_signed_int)
    17654    0.051    0.000    0.449    0.000 tools.py:82(read_utf8_like_unsigned_int)
   167206    0.405    0.000    0.405    0.000 {built-in method numpy.core.multiarray.empty}
    10891    0.006    0.000    0.260    0.000 objects.py:90(parse_data_field)
    15116    0.045    0.000    0.237    0.000 base.py:92(read_next_header)
        9    0.001    0.000    0.195    0.022 __init__.py:1(<module>)
    11111    0.006    0.000    0.172    0.000 tools.py:11(read_eventio_string)
39280/26460    0.026    0.000    0.162    0.000 base.py:222(seek)
    10891    0.010    0.000    0.118    0.000 objects.py:86(__init__)
     1910    0.009    0.000    0.117    0.000 _compression.py:66(readinto)
     1987    0.007    0.000    0.107    0.000 gzip.py:438(read)
    10892    0.005    0.000    0.103    0.000 tools.py:31(read_ints)

After:

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    160/1    0.002    0.000    3.927    3.927 {built-in method builtins.exec}
        1    0.018    0.018    3.927    3.927 time_it.py:1(<module>)
607931/371374    0.351    0.000    2.291    0.000 base.py:181(read)
1027123/675395    0.340    0.000    1.602    0.000 base.py:239(tell)
   214620    0.131    0.000    1.542    0.000 tools.py:20(read_from)
   676418    0.136    0.000    1.264    0.000 base.py:63(tell)
       70    0.066    0.001    1.165    0.017 objects.py:243(parse_data_field)
      129    0.001    0.000    1.130    0.009 objects.py:895(parse_data_field)
   677441    0.212    0.000    1.130    0.000 {method 'tell' of '_io._IOBase' objects}
      129    0.040    0.000    1.107    0.009 objects.py:932(_parse_list_type_1)
   704925    0.305    0.000    0.960    0.000 gzip.py:352(seek)
      129    0.000    0.000    0.595    0.005 objects.py:705(parse_data_field)
      129    0.000    0.000    0.589    0.005 objects.py:754(_parse_in_not_zero_suppressed_mode)
    96114    0.069    0.000    0.552    0.000 tools.py:6(read_array)
      129    0.533    0.004    0.533    0.004 {eventio.var_int.varint_arrays_differential_from_bytes}
44956/15116    0.021    0.000    0.525    0.000 search_utils.py:42(yield_all_objects_depth_first)
   373421    0.098    0.000    0.498    0.000 base.py:66(read)
    14359    0.031    0.000    0.474    0.000 base.py:208(__next__)
   704925    0.255    0.000    0.458    0.000 {method 'seek' of '_io.BufferedReader' objects}
   373422    0.151    0.000    0.401    0.000 gzip.py:271(read)
    17654    0.007    0.000    0.399    0.000 tools.py:51(read_utf8_like_signed_int)
    17654    0.023    0.000    0.392    0.000 tools.py:62(read_utf8_like_unsigned_int)
  1078347    0.205    0.000    0.296    0.000 _compression.py:12(_check_not_closed)
    10891    0.006    0.000    0.262    0.000 objects.py:91(parse_data_field)
    15116    0.044    0.000    0.235    0.000 base.py:92(read_next_header)
   704925    0.153    0.000    0.202    0.000 _compression.py:63(seekable)
        9    0.001    0.000    0.181    0.020 __init__.py:1(<module>)
    11111    0.006    0.000    0.173    0.000 tools.py:12(read_eventio_string)
39280/26460    0.026    0.000    0.164    0.000 base.py:222(seek)
375017/373499    0.070    0.000    0.150    0.000 {method 'read' of '_io.BufferedReader' objects}
    10891    0.009    0.000    0.118    0.000 objects.py:87(__init__)
    10892    0.005    0.000    0.103    0.000 tools.py:32(read_ints)

[maxnoe]

From the timings in #82, the next most expensive types are:

1. 2003 SimtTelCamOrgan
2. 2016 SimTelPixelTiming

Both contain loops over many iterations that read varints.

I will see if I can also optimize those using cython

[maxnoe]

I optmized 2016, total time went down from 2.8 to 2.4 seconds on my laptop for the sst1m file.

[dneise]

Let's merge this ... we can still add more stuff later.

[dneise]

Let's merge this we can still improve more stuff later in a dedicated PR ...

[maxnoe]

ok