Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update with use of esupy for flowlist storage #151

Merged
merged 14 commits into from
Nov 21, 2022
Merged

Conversation

bl-young
Copy link
Collaborator

@bl-young bl-young commented Nov 9, 2022

This implements the use of parquet storage via esupy for locally saving the flowlist and removes the flowlist parquet file from the repository

Attempting to access the flowlist will cause it to be created if not found locally.

@bl-young
Copy link
Collaborator Author

@WesIngwersen flagging you on this to ensure you are comfortable shifting to local data storage of the flowlist instead of within the repo.

@WesIngwersen
Copy link
Collaborator

@WesIngwersen flagging you on this to ensure you are comfortable shifting to local data storage of the flowlist instead of within the repo.

I assume it will store the flowlist parquet on the Data Commons?

@bl-young
Copy link
Collaborator Author

I can add the piece to download from data commons (I did not yet since generating locally is quite quick).
Would we prefer that by default users get the flowlist from data commons instead of generating locally?

@WesIngwersen
Copy link
Collaborator

I can add the piece to download from data commons (I did not yet since generating locally is quite quick). Would we prefer that by default users get the flowlist from data commons instead of generating locally?

Yes. We should go for consistency across other tools. where the principle API function to get the data in init calls in an esupy function to load from local storage or download it from the Commons if missing.

@bl-young
Copy link
Collaborator Author

ok added with c497ab6. I can confirm its working once we have a parquet up there.

@bl-young
Copy link
Collaborator Author

Assigning @catherinebirney for preparing data commons upon returning.

@bl-young bl-young self-assigned this Nov 21, 2022
@bl-young bl-young merged commit a0e473c into develop Nov 21, 2022
@bl-young bl-young deleted the esupy_storage branch November 21, 2022 18:56
@WesIngwersen
Copy link
Collaborator

tested and works for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants