Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Expose storage_options kwarg to readers/writers #6916

Closed
ayushdg opened this issue Dec 4, 2020 · 9 comments · Fixed by #12025
Closed

[FEA] Expose storage_options kwarg to readers/writers #6916

ayushdg opened this issue Dec 4, 2020 · 9 comments · Fixed by #12025
Assignees
Labels
cuIO cuIO issue feature request New feature or request improvement Improvement / enhancement to an existing function Python Affects Python cuDF API.

Comments

@ayushdg
Copy link
Member

ayushdg commented Dec 4, 2020

Is your feature request related to a problem? Please describe.
cuDF supports reading/writing to multiple data locations aided by fsspec. Reading data from remote source like cloud storage often requires additional kwargs passed to fsspec via the storage_options argument which is not exposed directly in cuDF but is a hidden kwarg.

Since v1.2, pandas has transitioned to using fsspec for a lot of file handling operations and now exposes the storage_options argument.

Describe the solution you'd like
cuDF can therefore expose this argument rather than using it as a hidden kwarg.

Describe alternatives you've considered
N/A

Additional context
Here are a couple of examples for pandas exposing storage_options from their docs: read_csv , to_parquet.

@ayushdg ayushdg added feature request New feature or request Needs Triage Need team to review and classify labels Dec 4, 2020
@ayushdg ayushdg added cuIO cuIO issue improvement Improvement / enhancement to an existing function labels Dec 4, 2020
@kkraus14 kkraus14 added Python Affects Python cuDF API. and removed Needs Triage Need team to review and classify labels Dec 7, 2020
@github-actions
Copy link

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

@github-actions github-actions bot added the stale label Feb 16, 2021
@ayushdg
Copy link
Member Author

ayushdg commented Feb 16, 2021

Can tackle once pandas requirements are updated (being worked on in #7375).

@github-actions github-actions bot removed the stale label Feb 16, 2021
@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@vuule
Copy link
Contributor

vuule commented Mar 22, 2021

The prerequisite PR has been merged.
@ayushdg, should I move this to 0.20?

@ayushdg
Copy link
Member Author

ayushdg commented Mar 22, 2021

Thanks for the ping @vuule. Pushing to 0.20 seems fine. Shouldn't impact workflows, since we already support it when passed via kwargs, but don't explicitly have it as an argument.

@chinmaychandak
Copy link
Contributor

chinmaychandak commented Mar 17, 2022

This will be really useful!

@ayushdg
Copy link
Member Author

ayushdg commented Mar 17, 2022

@chinmaychandak passing storage_options is already supported today. It just isn't exposed explicitly as a param, which this issue is meant to address.
Is storage_options not working for you for certain Io methods?

@chinmaychandak
Copy link
Contributor

No, I meant exposing this as an explicit param would be really useful. Not a blocker by any means.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuIO cuIO issue feature request New feature or request improvement Improvement / enhancement to an existing function Python Affects Python cuDF API.
Projects
None yet
6 participants