Fetch ESMValTool recipes in advance#582
Conversation
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 15 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
I think this is ready, but I'll have to have a more careful think. I'll get this in the next release |
Extract _resolve_cache_dir and _migrate_cache into DatasetRegistryManager
so registries can declare legacy cache directories. Files found at old
locations are moved to the new path on registration, avoiding
re-downloads after cache layout changes.
Wire up legacy_cache_dirs for ESMValTool datasets (previously flat under
os_cache("climate_ref")) and recipes (previously under
os_cache("climate_ref_esmvaltool")).
Also fix test_with_environment_variable for the empty-string env case.
lewisjared
left a comment
lewisjared
left a comment
There was a problem hiding this comment.
The change in directory will cause users to need to redownload the data again. Informing users that a refetch (ref providers setup --provider esmvaltool) might be easier than trying to migrate the old cache. What do you think?
|
It wasn't too bad to add a migration. This was required for the ilamb/pmp/sample-data registries. The added benefit is that only the actively used files were moved to the right directory making it easy to clean up dead files |
2 participants
Description
Closes #468
Checklist
Please confirm that this pull request has done the following:
changelog/