[C++][Parquet] Support writing lists that have null elements that are non-empty.

[asfimport]

With the new V2 level writing engine we can detect this case but fail as not implemented.  Fixing this will require changes to the "core" parquet API.

Reporter: Micah Kornfield / @emkornfield

Related issues:

• [C++][Parquet] Support reading FixedSizeLists when null values are present. (is related to)

_{Note: This issue was originally created as ARROW-8228. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
[~emkornfield@gmail.com] Can you elaborate on what's needed here?

The following is working at least:

>>> table = pa.table({"lists": [[1,2,None], None, [], [4,5]]})
>>> pq.write_table(table, "tt.pq")
>>> pq.read_table("tt.pq")
pyarrow.Table
lists: list<item: int64>
  child 0, item: int64
>>> pq.read_table("tt.pq").to_pandas()
             lists
0  [1.0, 2.0, nan]
1             None
2               []
3       [4.0, 5.0]

[asfimport]

Micah Kornfield / @emkornfield:
The case that isn't covered is if you have something like:

Null bitmap: [Not Null, Null, Not Null]

List_offset = [0, 2, 5, 7]

Data= [0, 1, 2, 3, 4, 5, 6]

 

Nullable FixedSizeList also runs into this issue (any nulls in the middle can't be written).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Example with fixed size list:

>>> arr = pa.array([[1, 2], None, [3, 4]], pa.list_(pa.int64(), 2))
>>> arr.values
<pyarrow.lib.Int64Array object at 0x7f64aeebb820>
[
  1,
  2,
  null,
  null,
  3,
  4
]

>>> pq.write_table(pa.table({"col": arr}), "test.parquet")
ArrowNotImplementedError: Lists with non-zero length null components are not supported

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
cc @paleolimbot