[air] Use custom fsspec handler for GS #24008
Conversation
| def create_dir(self, path, recursive): | ||
| try: | ||
| # No `create_parents` argument | ||
| self.fs.mkdir(path) |
There was a problem hiding this comment.
So it doesn't support create folders recursively now? What if the parent folder does not exist?
There was a problem hiding this comment.
No, the gs fsspec implementation does not accept a create_parents argument, which would be called by FSSpecHandler. As seen in the linked PR for gcsfs, it does not implement the fsspec base class correctly.
There was a problem hiding this comment.
GS does not have the notion of parent directories, everything is just an absolute path
There was a problem hiding this comment.
I see. Curious what is the difference between gs and gcs?
There was a problem hiding this comment.
There's no difference, these are the two registered URI protocols for google storage in fsspec: https://github.com/fsspec/filesystem_spec/blob/master/fsspec/registry.py#L107-L114
ericl
added
the
@author-action-required
krfricke
removed
the
@author-action-required
Why are these changes needed?
gcsfscomplains about an invalidcreate_parentsargument when using google cloud storage with cloud checkpoints. Thus we should use an alternative fs spec handler that omits this argument for gs.The root issue will be fixed here: fsspec/gcsfs#471
Related issue number
Checks
scripts/format.shto lint the changes in this PR.