Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Format] Enhance C-Data API to support non-cpu use cases #34971

Closed
zeroshade opened this issue Apr 7, 2023 · 1 comment · Fixed by #34972
Closed

[Format] Enhance C-Data API to support non-cpu use cases #34971

zeroshade opened this issue Apr 7, 2023 · 1 comment · Fixed by #34972
Assignees
@zeroshade
Labels
Component: Format Type: enhancement
Milestone
13.0.0

Comments

@zeroshade
Copy link
Member

Describe the enhancement requested

Currently the C-Data API assumes that all the buffers are CPU buffers and there is no way to pass device information through the ABI to allow using it for non-cpu memory allocations. We need to enhance the API to allow passing device and memory information without breaking the C-Data ABI, while also ensuring it is future-proofed for future developments with non-cpu devices.

Component(s)

Format
zeroshade added a commit to zeroshade/arrow that referenced this issue Apr 7, 2023
@zeroshade
apacheGH-34971: [Format] Enhance C-Data API to support non-cpu cases
3f9519d
@github-actions github-actions bot mentioned this issue Apr 7, 2023
Merged
@ianmcook
Copy link
Member

Also see this related discussion on the Arrow developer mailing list: https://lists.apache.org/thread/sntc3pp6msdvb94zhq2lvy70s1p6d1qg

zeroshade added a commit that referenced this issue Jun 6, 2023
@zeroshade @lidavidm
@pitrou @JohnZed @wgtmac
GH-34971: [Format] Add non-CPU version of C Data Interface (#34972)
105b9df 
### Rationale for this change
In order to support non-cpu devices and memory usage, we can add new `ArrowDeviceArray` and `ArrowDeviceArrayStream` structs to the C Data Interface in order to allow for handling these types of memory.

### What changes are included in this PR?
Definitions for a new `ArrowDeviceArray`, `ArrowDeviceArrayStream` and `ArrowDeviceType` enums.

* Closes: #34971

Lead-authored-by: Matt Topol <zotthewizard@gmail.com>
Co-authored-by: David Li <li.davidm96@gmail.com>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
Co-authored-by: John Zedlewski <904524+JohnZed@users.noreply.github.com>
Co-authored-by: Gang Wu <ustcwg@gmail.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
@zeroshade zeroshade added this to the 13.0.0 milestone Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zeroshade zeroshade

Labels
Component: Format Type: enhancement
Projects
None yet
Milestone
13.0.0
Development

Successfully merging a pull request may close this issue.

GH-34971: [Format] Add non-CPU version of C Data Interface zeroshade/arrow
2 participants
@ianmcook @zeroshade