[C++] Add data structure to "stage" a sequence of IPC messages from in-memory data

[asfimport]

Currently, when you need to pre-allocate space for a record batch or a stream (schema + dictionaries + record batches), you must make multiple passes over the data structures of interest (and use e.g. MockOutputStream to compute the size of the output buffer). It would be useful to make a single pass to "prepare" the IPC payload for both sizing and writing to prevent having to make multiple passes

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Original Issue Attachments:

• text.html

PRs and other links:

• GitHub Pull Request #1500

_{Note: This issue was originally created as ARROW-1860. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@xuepanchen I'm going to take a look at this if you don't mind. There are some nuances in the IPC writer around handling sliced bitmaps (see https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L125), and I want to see if we can improve the write performance in microbenchmarks while doing this refactoring

[asfimport]

Wes McKinney / @wesm:
I'm running into a slight issue with this refactoring around output streams which are currently at a position that is not a multiple of 8.

see https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/metadata-internal.cc#L903

This results in the entire message payload being non-deterministic depending on the state of the OutputStream. I don't think it will impact any applications, but I suggest we write padding bytes at the start to an 8-byte offset and then begin writing the message. This is the same approach used by WriteTensor in https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/writer.cc#L621.

@robertnishihara @pcmoritz @xhochy does this seem reasonable?

[asfimport]

Wes McKinney / @wesm:
Moving this to 0.10.0

[asfimport]

Wes McKinney / @wesm:
I will need to revisit this in the context of Flight RPC, where this data structure may be used to coordinate writes onto the GRPC output stream

[asfimport]

Wes McKinney / @wesm:
Resolved in db0ef22#diff-caf39e9e4747cfd6c97822e76dbac4e5