-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add filesystem abstraction using fsspec
#8379
Conversation
fsspec
Hey folks, Just in case it is not clear from the PR title. I would like to make sure the proposed API is agreed upon based on the presented cases before attempting to carry the fs abstraction throughout the rest of the code base. |
Changes made: * Add fs_utils.py for API that offers 1:1 replacement for current filesystem usage. * Update usages of open, glob, makedirs, normpath to use new API. * Update test/dataset/* to use in-memory filesystem, simplifying test and testing fsspec API.
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #8379 +/- ##
==========================================
- Coverage 88.32% 87.66% -0.66%
==========================================
Files 475 476 +1
Lines 28942 29011 +69
==========================================
- Hits 25562 25433 -129
- Misses 3380 3578 +198 ☔ View full report in Codecov by Sentry. |
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. I think this is a great start. I moved the files to torch_geometric.io
and suggest usage of
from torch_geometric.io import fs
I think this is a bit cleaner.
The cp
function for my taste seems a bit overloaded. Is it common practice to fuse downloading and extraction here? I think this is a bit error-prone. Made some updates but broke memory://
support in conftest
, since fs.ls
doesn't return the protocol :(
Changes made: