Support compound URLs

[martindurant]

With the cached FS and dask worker FS, not to mention zip, we have file-systems which take as input other file-systems or URLs. It would be convenient, but potentially confusing, to be able to address them with a single url:

"zip://s3://bucket/file.zip"

In practice, it may be tricky to differentiate between the parts of the path belong to which FS. This would probably need more syntax and maybe lead to more confusion than it clears up.

[ian-r-rose]

For what it's worth, fiona chooses a somewhat different syntax for compound urls with virtual filesystems:

"zip+s3://bucket/file.zip"

[martindurant]

Agree, but does the above give you one file or all of the files contained in the zip? If you include glob characters (for open_files), is that performed on s3 or in the zip?

[sgillies]

Does the existing URL chaining feature resolve this issue or is there something else to compound URLs @martindurant ?

The fiona library used by geopandas does have a different syntax, a syntax that was an attempt to improve on GDAL's chaining syntax. I have never enjoyed the idea proposed by GDAL that all zipfiles on S3 are virtually mounted on my filesystem under /vsizip/vsis3/. However, fiona's syntax is only cosmetic. Its Python classes rewrite, for example, zip+s3://... URIs to /vsizip/vsis3/... before passing them to GDAL functions.

I see fsspec's chaining, especially for longer and more complex chains, as better than fiona's, which shoehorns all of the protocols into one URI scheme. Whether parameters in a URL query string apply to the "s3" or the "zip" part of the scheme is undefined. I think you alluded to that above @martindurant. Whereas with fsspec's chaining, something like zip://bar.shp?key=val::https://example.org/foo.zip?key=val, a compact string encoding of the protocols and their options, may be feasible.

One thing that GDAL does well, and which I haven't seen yet in fsspec, is providing access to multiple-file data formats zipped together. Like the zipped-up shapefile (actually 3+ files), or a TIFF with georeferencing and other metadata.

[martindurant]

Yes, I will close this issue. Happy to continue the conversation here, though.

I must admit that I was wary of making my own syntax, and "mounting" into a current file-system tree certainly has a good pedigree. pyfilesystem2 support this, for instance, not to mention posix.

I think it's still fair to say that fsspec prefers kwargs to formatting arguments/queries into the URL itself, because it's much more flexible.

providing access to multiple-file data formats zipped together

Not quite sure what you mean. The following does work

fsspec.open("zip://*::s3://bucket/file.zip")

to create a list of open-files. You can choose to cache the original file or the decompressed files. However, you cannot glob for multiple zip files, since each is seen as a separate file system.

[mxmlnkn]

This definitely needs some documentation or did I miss something? The only mention I was able to find in the readthedocs was this line Also accepts compound URLs like zip::s3://bucket/file.zip and then I searched the Github code for zip:: and found this test, which fits my intended use case.

Found it: https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining
I probably was searching for compound instead of chain.

It is also not mentioned at all that this is necessary in the first place. My first try was tar://test.tar/bar. I was not surprised that this did not work because it would have to figure out via trial and error what parts of the path point to the archive and what parts point into the archive. However, my next test tar://test.tar was equally fruitless and failed with an unhelpful: IsADirectoryError: [Errno 21] Is a directory: error.

Furthermore, I tried to print-debug the arguments of fsspec.implementations.TarFileSystem and was surprised to see that it gets no arguments at all. This means it couldn't even try to find the given archive.

Bonus point: What about files/folders/paths containing :: in their file name?

[martindurant]

Certainly, would love to see improvements to the documentation between ZIP/TAR/libarchive and how they relate to one-another. ZIP came first, and some of its docstrings have moved upstream, but ended up in libarchive.

From your point of view, the salient thing to know, is that many FSs like these act on either another filesystem (e.g., DirFilesystem) or an open file (TAR). They take conventional arguments fs= and fo=, respectively, and in some cases target_protocol/target_options if are asking the filesystem to create a filesystem instance internally.
Chaining populates fs/fo arguments from the rest of the URL and infers arguments.

What about files/folders/paths containing :: in their file name?

This unfortunately fails with the top-level functions, but can still be made to work by instantiating the filesystems directly.