Permit composite URL protocol schemes in infer_storage_options

[amotl]

Hi there. In omniload, we are using composite URL protocol schemes like https+webdav://, mongodb+srv://, or mysql+pymysql://. While the code used custom auxiliary/helper functions before, we are now aiming to modernize and refactor a bit, for example towards using fsspec.utils.infer_storage_options across the board. On this occasion, we found that a specific regex pattern needed a minor adjustment to permit such URL protocol schemes that include a + character.

• Reference: Filesystem: Add read support for more fsspec-based filesystems (workbench) panodata/omniload#236

[martindurant]

What is fsspec supposed to do with there composite protocols, though - do they map onto omniload only and exactly?

Aside: omniload and intake should probably talk. Intake is caring less about writing yaml catalogues and pipelines, and (if anything) more about guessing file types and providing dataset summaries. Enumerating in detail all the things duckdb knows how to do, and providing a clear way to determine dataset invocation and summary all sound like good ideas.

[amotl]

Hi Martin,

thanks for your swift reply.

What is fsspec supposed to do with there composite protocols, though - do they map onto omniload only and exactly?

In our case, omniload is responsible for routing and dispatching of protocol schemes to dlt-based components that wrap the fsspec filesystem implementations. In this spirit, fsspec will probably not do anything specific to them today other than treating them opaquely, which seems to work well.

We just recently discovered infer_storage_options as a useful utility to streamline the downstream auxiliary logic around fsspec details in omniload, with that minor drawback. If it is not applicable or possible to adjust this utility function so it can be reused in omniload, we will either need to work around it or vendorize it.

Aside: omniload and intake should probably talk.

Thank you for quickly demonstrating ingredients of Intake. Indeed, that sounds nice. I've started a discussion thread so the conversation can grow beyond this specific PR discussion. 🙇

• omniload #2086

With kind regards,
Andreas.

[martindurant]

Please check the test failures: I think they are all cases where the protocol is omitted completely, which should result in the default "file", but the URL otherwise unchanged. These protocol-less URLs can contain some variants (not just "/...") due to windows.

[amotl]

Apologies. I made a mistake when refactoring the code block a bit to make it more readable. Should be fixed now.

[amotl]

Thank you for merging!