You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When having a binary array, converting that object dtype with to_pandas() (eg from converting a table to pandas) vs to_numpy() (or from calling np.asarray(..) on a pyarrow array) gives a considerable performance difference, although both are resulting in exactly the same numpy object dtype array (for to_pandas just wrapped in a pandas Series, but that should not give much overhead).
Profiling both showed a clear difference where there is a lot of usage of a hash table in the to_pandas() case, which reminded me that we have a deduplicate_objects option, which is False by default (which to_numpy uses) but set to True by default in to_pandas().
That explains the difference, and in a cases like this of all unique binary values, that just gives unnecessary overhead. Disabling it gives the expected similar performance for both to_numpu and to_pandas
In [4]: %timeit _ = arr.to_numpy(zero_copy_only=False)
375 ms ± 1.14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [5]: %timeit _ = arr.to_pandas(deduplicate_objects=False)
380 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Now I do wonder if we should consider turning it off by default in case of binary data .. (for strings, starting with pandas 3.0 which keeps arrow memory, the option will also not be relevant anymore)
When having a binary array, converting that object dtype with
to_pandas()
(eg from converting a table to pandas) vsto_numpy()
(or from callingnp.asarray(..)
on a pyarrow array) gives a considerable performance difference, although both are resulting in exactly the same numpy object dtype array (forto_pandas
just wrapped in a pandas Series, but that should not give much overhead).Example:
(noticed this in geopandas/geopandas#3322)
The text was updated successfully, but these errors were encountered: