[C++][Python] PyArrow Parquet writer fails writing [null] for null structs, but Datafusion works. Need to add [null] to current working [""] for NULLs.

[davlee1972]

Describe the bug, including details regarding any error messages, version, and platform.

I'm getting the following error trying to write a parquet file.

pyarrow.lib.ArrowInvalid: Column '_Id' is declare non-nullable, but contains nulls.

I think this is happening because there might be actual nulls in the column for the purpose of making a struct array even between the outer struct and inner struct. The image below is showing is_valid: [false] which looks odd to begin with..

The entire ShareClassType field is nullable and it contains a NULL, but the writer is complaining that a non-nullable field in belonging to non-nullable inner struct contains NULLs.

Here is the schema for this item:

and the data in it:

Component(s)

C++, Python

[davlee1972]

As a temporary work around.. I was able to write the pyarrow table using Datafusion..

Read it back using pyarrow and the rewrite it back as a rowgroup..

Datafusion changed the underlying data to [""].

Schema:

Data:

[davlee1972]

In PyArrow, a NULL column inside a StructArray is often initialized with a default value like "" (empty string) or 0. This happens because Arrow does not allocate null buffers for nested struct fields; instead, it puts a placeholder in the null slot.To fix this, use PyArrow compute functions to replace the empty strings with proper null values.

I guess using pa.null() or [None] to even out nested struct arrays won't work? I should be adding empty strings instead even though an empty string technically isn't a struct data type?

This could get complex since the outer struct has both scalar types and struct types..

I think this should be fixed in the parquet writer.. The [null] value appear to be perfectly valid in the pyarrow table..