Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Provide API for IPC roundtrip of RecordBatches not using the encapsulated message format #23120

Open
asfimport opened this issue Oct 3, 2019 · 2 comments

Comments

@asfimport
Copy link

Much of our IPC code presumes the use of the encapsulated message format. At some point we should ensure that we can address reconstruction of record batches when the "RecordBatch" Flatbuffer metadata was constructed more generally to reference buffers that may be located in different locations of a memory map or some other virtual address space

Reporter: Wes McKinney / @wesm

Note: This issue was originally created as ARROW-6783. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
What is the usefulness of this compared to the C data interface? Passing memory pointers in Flatbuffers wouldn't let you handle buffer lifetime, at least (unless you pass a release callback somewhere on the side).

@asfimport
Copy link
Author

Wes McKinney / @wesm:
I removed it from the 1.0.0 milestone. I agree that the C interface is a better solution for in-memory handoffs.

One meaningful deficiency with the C++ IPC implementation is the 0 frame of reference (relative to the start of the encapsulated message body). So it's only usable in the context of encapsulated messages. I'll rewrite the JIRA a bit to reflect this issue (which is non-urgent, but may be interesting to do something about someday)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant