feat(python): Make polars not copy data when importing from arrow #7084
Conversation
| @@ -140,7 +140,7 @@ def arrow_to_pyseries(name: str, values: pa.Array, rechunk: bool = True) -> PySe | |||
| elif array.num_chunks == 0: | |||
| pys = PySeries.from_arrow(name, pa.array([], array.type)) | |||
| else: | |||
| pys = PySeries.from_arrow(name, array.combine_chunks()) | |||
| pys = PySeries.from_arrow(name, array.chunks[0]) | |||
There was a problem hiding this comment.
I think we should do this only if array.num_chunks == 1 end leave the else branch as is.
There was a problem hiding this comment.
I'm happy to do that, but just to make sure we're in the same page, the existing if has next conditions:
array.num_chunks > 1array.num_chunks == 0else
Assuming num_chunks will always be an integer number greater or equal to zero, adding array.num_chunks == 1 will make that the else code is never executed.
Another option worth considering is having array.num_chunks == 0 and array.num_chunks == 1 and then have the rest as else instead of the current array.num_chunks > 1.
As said, happy to implement what you say, but just want to make sure we're understanding things the same way. Thanks
There was a problem hiding this comment.
Oh, sorry, my bad.
I didn't expand the lines above. You are right. 👍
ritchie46
changed the title
github-actions
bot
added
enhancement
|
Thanks! |
Closes #7007
Seems like when importing an Arrow array with one chunk, Polars is making a copy of the data by unnecessarily using
combine_chunks(), when it should use the chunk directly and not copy data.I didn't fine any test for the
array_to_pyseriesto write my test checking that memory is not being copied in the same file, so I usedtest_constructors.pywhich seemed the most appropriate. Let me know if it should live elsewhere.