[C++] Create a configurable implementation of RecordBatchReader that reads from Apache Parquet files

[asfimport]

This will be enabled by -ARROW-1008.-

A preliminary implementation of an arrow::RecordBatchReader was added in PARQUET-1166 but does not support configuring the batch size.  

Reporter: Wes McKinney / @wesm
Assignee: Hatem Helal / @hatemhelal

Related issues:

• [C++] GetRecordBatchReader in parquet/arrow/reader.h should be able to specify chunksize (is duplicated by)

PRs and other links:

• GitHub Pull Request #4304

_{Note: This issue was originally created as ARROW-1012. Please see the migration documentation for further details.}

[asfimport]

Hatem Helal / @hatemhelal:
It looks like PARQUET-1166 added a basic implementation of an arrow::RecordBatchReader that can be used to read from Parquet.  ARROW-3771 is a follow-up issue that I was looking into.

@wesm was there some additional functionality that you had in mind for this issue?

[asfimport]

Wes McKinney / @wesm:
@hatemhelal yes – the objective is to be able to configure a maximum record batch length (e.g. 64K, as an example) that obtain a stream of record batches no larger than that size. The GetRecordBatchReader returns an object that performs entire-row-group reads without configuration. This should be clarified in the JIRA description

[asfimport]

Wes McKinney / @wesm:
Moving this to the next release. I'd like to have at least a use case to drive this (the Datasets API should be that thing)

[asfimport]

Hatem Helal / @hatemhelal:
I added my use cases to the PR here:

#4304 (comment)

 

The primary use-cases for this API that I have come across are:

1. Iterative reading N rows a a time from a Parquet file. This might be necessary if the deserialized row group would be too large to fit in memory
2. Cheap-preview: with this change we can efficiently read the first N rows of a Parquet file.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4304
#4304