BUG: read_parquet can't detect column partitioning in non-local filesystems

[mvashishtha]

System information

• Modin version (modin.__version__): 86d3610

Parquet datasets are arbitrarily deep directories of data files that can include column partitioning (ref. here). In column partitioning, a directory like col1=3 only has data where column 1 is equal to 3. Modin tries to default to pandas for datasets with column partitioning, but it assumes that column partitioning can only happen when the path to read is a local directory. The directory could, for example, be on s3.

I think that nothing is functionally wrong when we fail to detect column partitioning. Modin can still read the data correctly. It's just not defaulting to pandas even though we expect it to.

[mvashishtha]

Maybe we should also consider not defaulting to pandas for cases like this. We can still parallelize the read across columns and might even do better than pandas in some cases.

[mvashishtha]

We can still parallelize the read across columns and might even do better than pandas in some cases.

Note that now that we parallelize across both rows and columns, I don't think we can read column-value-partitioned datasets so easily.

[mvashishtha]

I think that nothing is functionally wrong when we fail to detect column partitioning. Modin can still read the data correctly.

Per #5159, we now throw an error.

[mvashishtha]

@gshimansky could you please copy the partitioned directory s3://mahesh-vashishtha/modin_bug_5159_parquet/df.parquet to somewhere in s3://modin-datasets/ and make it publicly accessible so we can use that directory in a test case?

[gshimansky]

I created folder s3://modin-datasets/modin-bugs and copied modin_bug_5159_parquet into it. Please check that you have access, everything should be public.

[mvashishtha]

@gshimansky thanks, I can read the modin_bug_5159_parquet directory with no credentials, and I can list the contents in s3://modin-datasets/modin-bugs/.