BUG: for parquet files with list type columns, None values in such lists are not read correctly #603
Description
Relevant references:
- Noticed in this PR: BUG: fix support to read parquet files with list columns #597
- Reported upstream in this issue: Parquet: list field types with None values in the list give issues OSGeo/gdal#13448
- Solution/workaround provided in Arrow/Parquet: add a LISTS_AS_STRING_JSON=YES/NO open option OSGeo/gdal#13449
Todo, practically: in read_dataframe, when
use_arrow is False- GDAL > 3.12.1
- for files using the parquet or arrow driver
Use the new open option LISTS_AS_STRING_JSON=YES. When using this open option, list columns will be returned by GDAL as JSON, which are converted to lists again in read_dataframe, but then None values in those list will be correct.