-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(python)!: Use Object Store instead of fsspec for read_parquet
#13044
Conversation
a244340
to
c1483a7
Compare
0d819f8
to
242737c
Compare
read_parquet
instead of FSSPECread_parquet
to scan_parquet
internally
read_parquet
to scan_parquet
internallyread_parquet
to scan_parquet
internally
read_parquet
to scan_parquet
internallyread_parquet
to scan_parquet
internally
6862a8c
to
113e790
Compare
read_parquet
to scan_parquet
internallyread_parquet
Ahh, so I believe this is what broke my code here, where I open the file with fsspec filesystem and pass the data to read_parquet? print(pl.__version__)
fs = path.hook.filesystem
with fs.open(path.dataset_uri) as f:
test = pl.read_parquet(f)
print(test.height)
0.19.19
75594 After upgrading to 0.20.1:
It looks like the docs for read_parquet still show that file-objects are allowed though? source |
We are aware of the issue, see: A fix will come shortly. |
Ref #13040
Changes
read_parquet
toscan_parquet
where appropriate. This means that for purposes of cloud reading, it no longer uses fsspec.hive_partitioning
andretries
parameters.