From dask/dask#6047 (comment)
When specifying a directory to ParquetDataset, we will detect if a _metadata file is present in the directory and use that to populate the metadata attribute (and not include this file in the list of "pieces", since it does not include any data).
However, when passing a list of files to ParquetDataset, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).
We could detect it in a list of paths as well.
Note, I mentioned ParquetDataset, but if working on this, we should probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)
Reporter: Joris Van den Bossche / @jorisvandenbossche
Related issues:
Note: This issue was originally created as ARROW-8446. Please see the migration documentation for further details.
From dask/dask#6047 (comment)
When specifying a directory to
ParquetDataset, we will detect if a_metadatafile is present in the directory and use that to populate themetadataattribute (and not include this file in the list of "pieces", since it does not include any data).However, when passing a list of files to
ParquetDataset, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).We could detect it in a list of paths as well.
Note, I mentioned
ParquetDataset, but if working on this, we should probably directly do it in the datasets API-based version.Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)
Reporter: Joris Van den Bossche / @jorisvandenbossche
Related issues:
_common_metadatafor schema if_metadataisn't available (is related to)Note: This issue was originally created as ARROW-8446. Please see the migration documentation for further details.