[C++][Parquet] onvert flat SchemaElement vector to implied nested schema data structure

[asfimport]

To assist with conversion to in-memory nested data structures. Related: PARQUET-441

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• Add ParquetFilePrinter (is depended upon by)

_{Note: This issue was originally created as PARQUET-442. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
I'm going to take this on soon — there's some code in the outstanding patch for PARQUET-169 which I may be able to adapt for it. As part of it I'll need to address / write tests for PARQUET-441 and PARQUET-443

[asfimport]

Deepak Majeti / @majetideepak:
I hijacked PARQUET-169 to implement reading the repetition levels. We just read definition levels as of today.

[asfimport]

Wes McKinney / @wesm:
Cool – I'm planning to use the Impala test suite as a guide for writing tests for nested schema resolution so best to leave those bits alone (and the record data structures) for now. The schema will give us the information necessary to start working on the reassembly algorithms.

[asfimport]

Wes McKinney / @wesm:
I'm getting going on this, otherwise we're going to accumulate a bunch of refactoring debt with ColumnReader and its forthcoming unit tests (ColumnReader needs to accept an as-yet-unnamed ColumnSchema parameter containing the parquet::SchemaElement and the max rep/def levels computed by analyzing the schema tree).

[asfimport]

Wes McKinney / @wesm:
There's a lot more to do here than fits in one JIRA. I'll post a WIP patch within the next couple days and create a bunch of follow up JIRAs about schema-related stuff. The biggest priority is initializing the column readers with a proper ColumnDescriptor indicating the repetition / definition level of the leaf node (since we are unable to read most nested data at the moment).

[asfimport]

Deepak Majeti / @majetideepak:
Another priority is a column_id indicating the physical column number(parquet::SchemaElement with 0 children). Is your plan to implement the Impala's way, which is to build a tree from the flat parquet::SchemaElement array?

[asfimport]

Wes McKinney / @wesm:
Yep, in progress here: https://github.com/wesm/parquet-cpp/tree/PARQUET-442. Will cc you when I've written enough tests to solidify the main APIs

[asfimport]

Wes McKinney / @wesm:
Patch in progress: apache/parquet-cpp#38. Going to add some more tests but looking for reviews from experienced Parquet people (like to avoid any pitfalls with schema analysis early if possible).

[asfimport]

Julien Le Dem / @julienledem:
Issue resolved by pull request 38
apache/parquet-cpp#38