Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review if we can use fsspec for data loading #1462

Open
roll opened this issue Mar 13, 2023 · 4 comments
Open

Review if we can use fsspec for data loading #1462

roll opened this issue Mar 13, 2023 · 4 comments
Labels
general General improvements

Comments

@roll
Copy link
Member

roll commented Mar 13, 2023

Overview

https://pypi.org/project/fsspec/

@roll roll added the general General improvements label Mar 13, 2023
@davidgasquez
Copy link

Heya @roll! I'm curious if you can expand on this one. Are you thinking on using fsspec to replace the scheme parts of the codebase?

I've been thinking on the best way to read/write data on a non existing scheme and thinking about this these days.

@roll
Copy link
Member Author

roll commented Mar 23, 2023

@davidgasquez
Yes, we historically used these loaders approach created for tabulator, but I'm curious if fsspec can handle it (I haven't dove into it yet unfortunately)

@davidgasquez
Copy link

I think it should!

A potentiall intermediary approach might be to add a fsspec scheme via plugins. Does this makes sense?

@roll roll added this to the v6 milestone Mar 27, 2023
@roll
Copy link
Member Author

roll commented Mar 27, 2023

TBH, currently, I'm not sure as it seems fsspec might just a perfect replacement for the whole Loaders part making the parsers and the framework way simpler.

We're going to work soon on the v6 - https://github.com/orgs/frictionlessdata/projects/16/views/11 - which will include optimizations for the whole reading flow. Might be a good chance for the fsspec migration

@roll roll removed this from the v6 milestone May 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
general General improvements
Projects
Status: No status
Development

No branches or pull requests

2 participants