[C++] Create Protocol Buffers interface for iterating over the semantic "rows" of a record batch, and accessing the rows using the protobuf API

[asfimport]

The desired workflow:

• User writes a .proto file describing the structure of a "row" as a Message

• Given the generated pb.h bindings, an Arrow users can iterate over an arrow::RecordBatch, each iteration populating an instance of the Row message

• The values of the row can then be accessed via the standard Protobuf APIs

  A corresponding interface could be developed to write a RecordBatch using protobufs as input, but that could be its own project

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-3151. Please see the migration documentation for further details.}

[asfimport]

Paul Rogers:
Let's see if we can coordinate on this. I'm starting work on a proposal for a "RowSet" interface to be ported over from Drill that provides a simple row-based API to read from, and write to, vectors. On the write site, the mechanism also enforces memory limits, which is the key reason Drill created the "RowSet" abstraction.

Given that this project will need a way to assemble a row from a bundle of vectors, the "columnar-to-row" mechanism of RowSet might be a way to populate the row buffer.

On the other hand, the RowSet code from Drill is in Java, this is C+. Still, might make sense to port the mechanism to C+ so it can be used in multiple contexts.

Any background docs I could read to get a better understanding of the project context to determine if what was just said above makes sense in this context? Thanks.

[asfimport]

Wes McKinney / @wesm:
Thanks Paul – indeed this is a C++ initiative but I agree it would be useful to align on some target use cases, requirements, etc. What I intended here is pretty orthogonal to memory management, or any details of where the Arrow columnar memory comes from; a user might have some code which utilizes a Protobuf-based row interface and we want to feed that Arrow data as quickly as possible

[asfimport]

Antoine Pitrou / @pitrou:
This is a sizable piece of work and no activity has occurred for the last years. Closing.