Unable to read hive-style partitioned parquet file using read_parquet

[lmocsi]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

"""
I have a partitioned parquet file like this (filename is
my_table_name, and it is partitioned by calendar_date column, see attached):

my_table_name
 |
 +[calendar_date=2023-08-01 00%3A00%3A00]
 | |
 | + part-00000-b3b03a4e.c000.snappy.parquet
 | 
 +[calendar_date=2023-08-02 00%3A00%3A00]
 | |
 | + part-00000-f4b5a541.c000.snappy.parquet
 |
 +[calendar_date=2023-08-03 00%3A00%3A00]
   |
   + part-00000-6cf29fe7.c000.snappy.parquet

"""

import polars as pl
df = pl.read_parquet(path+ 'my_table_name/*')  # on SO it was recommended, that /* could be used

"""
It is giving me the error:
ComputeError: error while reading /path/my_table_name/CALENDAR_DATE=2023-08-01 00%3A00%3A00: External format error: File out of specification: underlying IO error: Invalid argument (os error 22)
"""
my_table_name.zip

Issue description

It seems, that as of now polars is supporting only two types of parquet structure:

1. all the data is in one parquet file
2. the data is split into separate files within one directory

Though, true partitioned parquet files (where you have separate directory for each partition) does not seem to be supported. :(

Expected behavior

Able to read the partitioned parquet file, even as a lazy dataframe (just like Spark does)

Installed versions

--------Version info---------
Polars:              0.18.11
Index type:          UInt32
Platform:            Linux-4.18.0-372.51.1.el8_6.x86_64-x86_64-with-glibc2.28
Python:              3.9.13 (main, Oct 13 2022, 21:15:33) 
[GCC 11.2.0]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
cloudpickle:         2.0.0
connectorx:          <not installed>
deltalake:           <not installed>
fsspec:              2022.02.0
matplotlib:          3.7.2
numpy:               1.21.6
pandas:              2.0.3
pyarrow:             12.0.1
pydantic:            <not installed>
sqlalchemy:          1.4.27
xlsx2csv:            <not installed>
xlsxwriter:          3.1.2

</details>

[cmdlineluser]

There is scan_pyarrow_dataset()

import pyarrow.dataset as ds

pl.scan_pyarrow_dataset(ds.dataset("my_table_name")).collect()

# shape: (13, 2)
# ┌─────────┬─────────┐
# │ USER_ID ┆ TRX_CNT │
# │ ---     ┆ ---     │
# │ f64     ┆ f64     │
# ╞═════════╪═════════╡
# │ 1000.0  ┆ 434.0   │
# │ 1001.0  ┆ 11.0    │
# │ 1002.0  ┆ 3.0     │
# │ 1003.0  ┆ 555.0   │
# │ …       ┆ …       │
# │ 1001.0  ┆ 21.0    │
# │ 1003.0  ┆ 44.0    │
# │ 1005.0  ┆ 111.0   │
# │ 1008.0  ┆ 222.0   │
# └─────────┴─────────┘

[alexander-beedie]

df = pl.read_parquet(path+ 'my_table_name/') # on SO it was recommended, that / could be used

This is correct if all the files are in the same directory, but otherwise (as @cmdlineluser says) you need to use scan_pyarrow_dataset to read directory-nested (hive-style) partitioned parquet data.

(I've just committed a small update to the docs that adds a more explicit note to read_parquet and scan_parquet to help direct users to the right method).

[stinodego]

I think read_parquet should support this. If I am trying to read a parquet file, I should be using read_parquet. Even if it is partitioned.

Not sure how hard this is to implement, but it should be a goal, in my opinion.

[lmocsi]

Could pl.read_parquet() just call this pl.scan_pyarrow_dataset() function?
On the other hand, the reading up this parquet should include the partitioning column (here calendar_date), as well...

[alexander-beedie]

(Updated the title to clarify the issue more specifically, so we can reference it easily later).

[lmocsi]

Added my last comment as a separate issue: [https://github.com//issues/10296]

[universalmind303]

looks like #4347 and #426 are duplicates. Since this is the most recent one, I'll keep this open & close out the other two.

[ddutt]

I think read_parquet should support this. If I am trying to read a parquet file, I should be using read_parquet. Even if it is partitioned.

Not sure how hard this is to implement, but it should be a goal, in my opinion.

scan_pyarrow_dataset performance isn't as good. Im doing the addition of the hive directories as columns, and i have a lot of folders, and this method is still faster than using scan_pyarrow_dataset. Please add this support to scan parquet and read_parquet

[stinodego]

This should be fixed by #13044

If not, please comment and I can reopen this issue.

[lmocsi]

Hi, In polars 0.20.5 this works: pl.read_parquet('my_table_name/**/*.parquet') but this: pl.read_parquet('my_table_name') throws the bellow error: IsADirectoryError: expected a file path; 'my_table_name' is a directory Can I not use a directory here? (I should be able to) Thanks, Lmocsi <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Vírusmentes.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

…

On Fri, Jan 12, 2024 at 11:43 PM Stijn de Gooijer ***@***.***> wrote: This should be fixed by #13044 <#13044> If not, please comment and I can reopen this issue. — Reply to this email directly, view it on GitHub <#10276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APOYDWZQGT2KDV76BDFLRDLYOG4ATAVCNFSM6AAAAAA3DB4GP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJQGA3TONZUGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>