Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Error reading parquet from s3 with s3fs >= 0.3.0 #27756
Code Sample, a copy-pastable example if possible
import pandas as pd df = pd.read_parquet('s3://my-bucket/df.parquet')
In version 0.3.0 s3fs removed the
Should pandas check the s3fs version and call the right attribute based on that?
Indeed this is an API change. However, I am surprised that anyone is opening a file and then using the FS methods of the attribute of that file - you presumably have the FS available directly anyway at this point.
Indeed, rather than test specifically for s3 URLs, I would strongly encourage pandas to use fsspec directly, so that then you can read from any of the implementations supported by fsspec.
I'm not sure what's best.…
On Mon, Aug 5, 2019 at 9:58 AM Chris Stadler ***@***.***> wrote: Perhaps there should be a function returning both the file and the filesystem, which can be used here instead of get_filepath_or_buffer. That would avoid S3File.s3/S3File.fs. If that sounds like a reasonable direction I will work on a PR. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27756?email_source=notifications&email_token=AAKAOIX27VNYLVWZZADDDFTQDA5Z3A5CNFSM4IJLDNJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3SCWKI#issuecomment-518269737>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIX6R6HBTG6K5TWDRYLQDA5Z3ANCNFSM4IJLDNJQ> .
For the interested, the implementation is as simple as
--- a/fsspec/spec.py +++ b/fsspec/spec.py @@ -1154,6 +1154,9 @@ class AbstractBufferedFile(io.IOBase): def __str__(self): return "<File-like object %s, %s>" % (type(self.fs).__name__, self.path) + def __fspath__(self): + return self.fs.protocol + "://" + self.path +