ARROW-89: [Python] Add benchmarks for Arrow<->Pandas conversion

[xhochy]

No description provided.

[xhochy]

Seems like we only have zero-copy Pandas-to-Arrow for int64 columns at the moment as float64 seems to have some overhead:

[ 42.86%] ··· Running array.PandasConversions.time_from_series                                                                               ok
[ 42.86%] ···· 
               ========== ========== ========== ==========
               --                      dtype              
               ---------- --------------------------------
                  size      int64     float64      str    
               ========== ========== ========== ==========
                   1       197.17μs   197.98μs   201.56μs 
                 100000    198.50μs    2.04ms     9.51ms  
                1000000    196.28μs   17.12ms    96.38ms  
                10000000   195.93μs   165.74ms    1.07s   
               ========== ========== ========== ==========

[wesm]

@xhochy yes -- because integer data in pandas does not support nulls, the null accounting can be skipped. In the case of float64, you must convert the NaN values to a bitmap. See the ValuesToBitmap function.

[wesm]

Not sure if you are benchmarking -O3 release builds but that should make a performance difference.

[xhochy]

I'm benchmarking -O3 builds, I've got a separate build env locally for these benchmarks ;)

[wesm]

I figured =)

[xhochy]

Also made me a VM setup to run the benchmarks independently from my current setup in a VM: https://github.com/xhochy/arrow-performance-setup Maybe it could be of use for someone else, at least for me, I can be sure that those builds are all the ones with -O3.

[wesm]

Ah, that is super useful. I starred it for now, let's see how things develop, and having reproducible testing VMs (we could use docker, too?) will be very helpful, I'm sure.

[wesm]

+1, thank you