removing fsspec in python in favour of object_store in rust

[svaningelgem]

Description

Hi @ritchie46 ,

I was in discussion with @Qqwy about the following:
Within rust we have now the *_cloud methods for sinking to a cloud service via object_store.
Is it an idea to generalize every call to the read/scan/write methods to make use of object_store instead of relying (on the python side) on fsspec?

My idea is this:

• normal methods: accept str | Path | BytesIO (writable bytestream for write, readable for sink/scan)
• cloud methods: accept str | Path (like S3Path) | BytesIO (writable bytestream for write, readable for sink/scan) + cloud options

Within the Python API I would combine these 2 with a default parameter for the cloud options that could optionally be passed in for cloud paths. Sadly the same is not possible in Rust.

My only issue was with globs: how to handle these, but it seems (Qqwy checked it) that these are handled by object_store as well. So we're fine there.

Second what Qqwy brought up was that there are methods like parse_url that parses a given uri tries to figure out what you're trying to do. [And relative paths are not supported].
==> For the parse_url & glob logic he concluded that it's already present in polars/object_store.

My ultimate wish for the "file interface api" would be that there is struct that accepts a list of writable/readable streams and does whatever it needs doing for the specific filetype.
If that's done, then any kind of file format could be easily implemented as the ability to read/write from a stream. All the other code would be boilerplate.

My question for this ticket is: what is your view on the API?
(and secondary: is this überhaupt possible in rust?)

Thanks

[ion-elgreco]

Why not have one method, and based on that you invoke the object store? Because object_store can also be used for local files, same as fsspec.

[svaningelgem]

Even better. I just don't know the capabilities of rust :'(

[abealcantara]

Agree, I vote for merging sink_parquet_cloud and sink_parquet having a single method that supports both local files and object store files in Rust for each format and read/write operations feels more natural and similar to what is currently available in other data processing frameworks. As @svaningelgem pointed out, the object_store crate supports Path and it also supports local storage, so I think it will simplify the Polar's API. We can then implement cloud support for the other formats without adding more methods to the DataFrame API.

[ritchie46]

I got this on my todo list. Will look into this a bit later.

[Qqwy]

My only issue was with globs: how to handle these, but it seems (Qqwy checked it) that these are handled by object_store as well. So we're fine there.

For correctness: These are actually handled by Polars itself and not by ObjectStore. c.f. polars-io/src/cloud/gloub.rs.

Why not have one method, and based on that you invoke the object store? Because object_store can also be used for local files, same as fsspec.

I don't know fsspec in detail, but ObjectStore only handles absolute paths, and local paths need to be prefixed by file://. We thus need to take a little care to translate relative paths to file:// absolute paths beforehand.

[kylebarron]

People here might be interested: I wrote obstore, a binding of the Rust object_store crate for Python. Early benchmarks indicate up to 10x higher throughput than s3fs when used in an async Python context (this specific scenario is a range request fetching the first 16kb of a file).

Is polars already using fsspec in an async way?

If you use obstore, it would still go through Python, and so may be less efficient than a native implementation in Rust using object_store, but it also means you wouldn't have to link object_store at compile time.

[clbarnes]

You could also consider openDAL, another rust-first Apache project for abstracting over data stores. OpenDAL is more flexible in some ways and supports many more stores, although I think it's still missing "choose store by URL": https://crates.io/crates/opendal