[Python] pandas compat: clean-up code in columns Index reconstruction in Table.to_pandas #40720
Comments
jorisvandenbossche
added a commit
to jorisvandenbossche/arrow
that referenced
this issue
Mar 21, 2024
… column names in Table.to_pandas
jorisvandenbossche
added a commit
that referenced
this issue
Mar 26, 2024
…n names in Table.to_pandas (#40721) ### Rationale for this change The `pandas_compat.py` has over the years grown quite complex and a lot of pandas compatibility code, which probably can be simplified nowadays because of not supporting old pandas and Python versions anymore. One part of the code where this is the case is in the reconstruction of the `.columns` Index object of the resulting DataFrame. Right now that always goes through a MultiIndex (even for simple column names), which has quite some overhead of the simple case. And it also has some old Python/pandas compat code that could be removed. ### What changes are included in this PR? The simplification to not go through a MultiIndex for the simple cases gives a nice speed-up as well: ```python In [1]: table = pa.table({'a': [1, 2, 3], 'b': [0.1, 0.2, 0.3], 'c': [3, 4, 5]}) In [2]: %timeit table.to_pandas() 251 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) # <-- main 68.1 µs ± 894 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) # <-- PR ``` ### Are these changes tested? We should have extensive existing tests for this ### Are there any user-facing changes? That should not be the case * GitHub Issue: #40720 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
|
Issue resolved by pull request 40721 |
|
I tried to find out what the supported versions of pandas/python are, but couldn't find anything. It would be great if you could point me somewhere @jorisvandenbossche. I noticed that Pyarrow 16.1.0 seems not to be usable with pandas 1.5.3 anymore, for example (it was I think until 16 dropped). I'm getting |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
pandas_compat.pyhas over the years grown quite complex and a lot of pandas compatibility code, which probably can be simplified nowadays because of not supporting old pandas and Python versions anymore.One aspect that I noticed while working on
to_pandaswas the complexity in the reconstruction of the.columnsIndex object of the resulting DataFrame. Right now that always goes through a MultiIndex (even for simple column names), which has quite some overhead of the simple case.The text was updated successfully, but these errors were encountered: