Catalogs refactor

[DirkEilander]

Issue addressed

Fixes #844

Explanation

This PR updates the predefined catalogs of HydroMT. The catalogs are now stored in their own folder with each version having its own data catalog. The versions are tracked in a registry.txt. This file contains the relative path and the hash of the data catalog.
Before the different versions of datacatalogs were retrieved by using the git hashes stored in predefinend_catalogs.yml. This file has henceforth been deprecated. Instead a PredefinedCatalog class has been created to retrieve specific versions of predefined data catalogs.

Example of retrieving different versions of data catalogs:

from hydromt import DataCatalog

data_catalog = DataCatalog(data_libs="deltares_data=v0.5.0") # Retrieve deltares_data v0.5.0 data catalog
data_catalog_latest = DataCatalog(data_libs="deltares_data") # Will implicitly retrieve the latest version of deltares_data

If you want to create your own version of deltares_data for example, you should make a new folder in the data/catalogs/deltares_data/ with the version as its name and put the data catalog file there and call name it data_catalog.yml. If you then run the update_versions.py script to update the registry file.

Checklist

• Updated tests or added new tests
• Branch is up to date with main
• Tests & pre-commit hooks pass
• Updated documentation if needed
• Updated changelog.rst if needed
• For predefined catalogs: update the catalog version in the file itself, the references in data/predefined_catalogs.yml, and the changelog in data/changelog.rst

Additional Notes (optional)

Add any additional notes or information that may be helpful.

[DirkEilander]

TODO:

• try to fix linux vs windows os issue with hash; if not easily fixed remove it
• add older version to artifact_data and deltares_data (check if version numbering makes sense)
• docs and changelogs
• cache data_catalog.yml files in DataCatalog._cache_dir/<catalog_name>//data_catalog.yml
• if versions.yml not accessible use predefined_catalogs._get_catalog_versions to browse local versions

Decided to postpone to v1

• create entrypoint for data catalogs plugins

[Tjalling-dejong]

@DirkEilander in the case you are still working, I do not really understand what I need to mock with the entry_points in _get_catalog_eps. The entry_points are empty in the tests and where is hydromt.catalogs entrypoint defined?

[DirkEilander]

@DirkEilander in the case you are still working, I do not really understand what I need to mock with the entry_points in _get_catalog_eps. The entry_points are empty in the tests and where is hydromt.catalogs entrypoint defined?

@Tjalling-dejong The idea is to test predefined catalogs which are provided by entrypoints. These can be set by users in their pyproject.toml as a "hydromt.catalogs" entrypoint and are discovered in predefined_catalogs.py. For the core these are bypassed as LOCAL_EPS.

I'm however also fine with skipping this test for now as @savente93 has done a really nice job in v1 to generalize all entrypoints in plugins.py. It would be nice to use that after merging this work in v1 too in which case the tests will probably have te be rewritten. Compared to the other entrypoints this refers to a versions.yml file rather than a class though. But if needed we could refactor the predefined catalog versions into a simply (pydantic) class as well.

[savente93]

@Tjalling-dejong if you want me to go over how we did plugins.py let me know

[savente93]

@Tjalling-dejong can you please update the title, description and checklist?

[savente93]

I'll have to finish the full review later, but here are some preliminary comments.

[savente93]

Can you add an explanation on how to call this script? e.g. does it take arguments? if so which?

[savente93]

could you use SHA-1? that's what git uses which I think is more consistent.

[Tjalling-dejong]

@savente93 Can you review this PR?
Note that the Build Documentation fails because it tries to retrieve a data catalog versions.yaml from main that does not exist yet.

[savente93]

@Tjalling-dejong I'll have a more indepth look in a bit, the failing docs are okay, but there are still a few issues flagged my sonar cube and the PR description is still empty. Could you fix those in the mean time please

[savente93]

So, perhaps a stupid question but: is this change backwards compatible? i.e. will previous versions of hydromt break if we merge this? (I'll do a review regardless but I'd like some confirmation that this won't break things again before we merge)

[Tjalling-dejong]

So, perhaps a stupid question but: is this change backwards compatible? i.e. will previous versions of hydromt break if we merge this? (I'll do a review regardless but I'd like some confirmation that this won't break things again before we merge)

I dont think it will break previous versions of hydromt. It works even if you do not specify a data catalog version.

[savente93]

Nice work Tjalling! I like the descriptions and testing. Just a couple small comments left and then it should be good to go :)

[savente93]

LGTM but Jaap said he wanted to review as well so I'll leave the final approve to him