Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add selective zero-copy DataFrame interchange #6951

Closed
cnpryer opened this issue Feb 16, 2023 · 3 comments · Fixed by #10262
Closed

Add selective zero-copy DataFrame interchange #6951

cnpryer opened this issue Feb 16, 2023 · 3 comments · Fixed by #10262
Assignees
Labels
accepted Ready for implementation enhancement New feature or an improvement of an existing feature

Comments

@cnpryer
Copy link
Contributor

cnpryer commented Feb 16, 2023

I thought I'd open this issue to surface the interest and serve as a place to coordinate if necessary.

The ask is to start opening up the protocol usage for any use-cases that are zero-copy compatible.

As it stands, if I understand, the state of converting between Pandas and Polars is:

Cannot utilize from_dataframe in a zero-copy manner. This will throw a NotImplementedError. We can refine this later by detecting which dtypes are present in the dataframe and determining if those can be handled in a zero-copy manner.
- #6581

I looked at this stuff a while ago because this feature specifically interests me. So I generally understand the level of effort for this ask. I'm in a spot where I'd like to migrate most of our Pandas usage to Polars. I've visited this topic a few times in the past year. The conversation is much easier with zero-copy options both ways. So I'd be very excited to see this feature implemented wherever it can be.

@cnpryer cnpryer added the enhancement New feature or an improvement of an existing feature label Feb 16, 2023
@stinodego
Copy link
Member

I think @ghuls added this recently in #6844

We can probably still improve!

@cnpryer
Copy link
Contributor Author

cnpryer commented Feb 17, 2023

Of course @ghuls did lol. Such a machine. That's awesome.

I plan to migrate a lot of stuff to polars over the next month, so I'll give it a spin.

@cnpryer cnpryer closed this as completed Feb 17, 2023
@stinodego
Copy link
Member

stinodego commented Feb 17, 2023

I am re-opening this as I don't think the current from_dataframe implementation guarantees zero copy in the right way. See my comment here.

Let's see how this turns out - I think adding the proper safeguards to guarantee zero-copy is actually not too big a hurdle. I don't mind doing some work there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation enhancement New feature or an improvement of an existing feature
Projects
Archived in project
2 participants