Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate interop functionality to separate module #15310

Closed
vyasr opened this issue Mar 14, 2024 · 0 comments · Fixed by #15325
Closed

Migrate interop functionality to separate module #15310

vyasr opened this issue Mar 14, 2024 · 0 comments · Fixed by #15325
Labels
feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Comments

@vyasr
Copy link
Contributor

vyasr commented Mar 14, 2024

Currently pylibcudf Table and Scalar support [to|from]_arrow methods. This is an antipattern in pylibcudf where essentially all functionality is provided via free functions operating on objects rather than via class methods. Furthermore, the current approach forces using pyarrow Cython in scalar.pyx and scalar.pxd, which in turn essentially means that every part of cudf uses arrow Cython. As #15193 progresses we will also be looking to isolate the pyarrow dependency entirely from pylibcudf and probably make it optional (cudf will still require it), so it would be best to put all such logic directly into the interop module. The exact implementations will change as we migrate to a capsule-based FFI rather than the current approach of passing typed arrow objects directly (which requires linking to arrow).

While making this change, we should also update the scalar converter to not require column metadata. We should be able to spoof that since it is only really necessary to create the intermediate column and is discarded immediately afterwards.

@brandon-b-miller brandon-b-miller added feature request New feature or request Python Affects Python cuDF API. labels Mar 18, 2024
rapids-bot bot pushed a commit that referenced this issue Mar 18, 2024
Resolves #15310. Contributes to #15193

In addition, this PR adds pylibcudf.Column<-->pyarrow.Array interconversion as a benefit

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: #15325
@vyasr vyasr added the pylibcudf Issues specific to the pylibcudf package label May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
Archived in project
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants