Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] rework data loader module, ability to specify download mirrors #4985

Merged
merged 8 commits into from
Aug 12, 2023

Conversation

fkiraly
Copy link
Collaborator

@fkiraly fkiraly commented Jul 30, 2023

This PR reworks the somewhat messy data loader module, and adds an ability to specify download mirrors for remote datasets.

This is towards #4754 but does create a framework for arbitrary data loader mirrors - I focused mainly on enabling the mirroring/fallback feature for existing loaders, rather than a framework level refactor.

A refactor would also include the forecasting data loader and factor out more of the repetitive code in functions.

FYI @achieveordie, @hazrulakmal, as you have both recently worked on this module.

In terms of addressing #4754, it should now be easy to add an arbitrary number of mirrors to prevent a blowout failure to users if the UEA data repository decides to abruptly change folder structure again without deprecation or warning.

@fkiraly fkiraly added the module:datasets&loaders data sets and data loaders label Jul 30, 2023
@fkiraly fkiraly merged commit 0733329 into main Aug 12, 2023
24 checks passed
@fkiraly fkiraly deleted the clean-dataload-and-cache branch August 12, 2023 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module:datasets&loaders data sets and data loaders
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant