Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support creating arrays from externally owned memory like Vec or String #1516

Closed
jhorstmann opened this issue Apr 1, 2022 · 0 comments · Fixed by #1494
Closed

Support creating arrays from externally owned memory like Vec or String #1516

jhorstmann opened this issue Apr 1, 2022 · 0 comments · Fixed by #1494
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@jhorstmann
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I'd like to process data that is coming from outside arrow using arrow compute kernels without copying the data.

The FFI api does not seem to support this use case at the moment. The only way I found to create an FFI_ArrowArray was with an already existing ArrayData struct. Creating ArrayData requires at least one Buffer, and creating a Buffer from foreign memory again needs an FFI_ArrowArray.

Describe the solution you'd like

The Buffer already supports two ways of deallocating memory.

  • Via arrows cache-aligned allocator
  • With a callback in an FFI_ArrowArray

The second option can be generalized so that the callback can be any type using a dyn trait.

Describe alternatives you've considered

An alternative could be to make it easier to create an FFI_ArrowArray from a single memory region, but that seems to be more complex and also less general.

Additional context

This functionality could also allow zero-copy conversion from arrow2.

@jhorstmann jhorstmann added the enhancement Any new improvement worthy of a entry in the changelog label Apr 1, 2022
@alamb alamb added the arrow Changes to the arrow crate label Apr 15, 2022
@alamb alamb changed the title Allow creating buffers from externally owned memory like Vec or String Support creating arrays from externally owned memory like Vec or String Apr 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants