Add "grid" as a tag to the output file template for CMIP6 #1356
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1356 +/- ##
=======================================
Coverage 88.26% 88.26%
=======================================
Files 194 194
Lines 9836 9836
=======================================
Hits 8682 8682
Misses 1154 1154 Continue to review full report at Codecov.
|
| @@ -28,7 +28,7 @@ CMIP6: | |||
| ETHZ: '{exp}/{mip}/{short_name}/{dataset}/{ensemble}/{grid}/' | |||
| SYNDA: '{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{latestversion}' | |||
| input_file: '{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc' | |||
| output_file: '{project}_{dataset}_{mip}_{exp}_{ensemble}_{short_name}' | |||
| output_file: '{project}_{dataset}_{mip}_{exp}_{ensemble}_{short_name}_{grid}' | |||
There was a problem hiding this comment.
I think this needs a bit more attention: is grid now a mandatory parameter? If it is, we need to add an exception in _recipe.py in the form of a RecipeError if it's not provided
There was a problem hiding this comment.
I think for CMIP6 grid has always been mandatory. Do you have a recipe that runs without a specified grid?
There was a problem hiding this comment.
it's mandatory while data finder runs, but this can be promoted to before that via a RecipeError rather than having the run crash with Missing data when no grid is specified:
2021-10-12 12:12:21,985 UTC [10195] ERROR Could not create all tasks
2021-10-12 12:12:21,985 UTC [10195] ERROR Missing data for preprocessor validation_with_ERA-Interim/tas:
- Dataset key 'grid' must be specified for {'preprocessor': 'pp_rad', 'variable_group': 'tas', 'short_name': 'tas', 'diagnostic': 'validation_with_ERA-Interim', 'dataset': 'IPSL-CM6A-LR', 'project': 'CMIP6', 'mip': 'Amon', 'exp': 'historical', 'ensemble': 'r1i1p1f1', 'start_year': 1900, 'end_year': 2000, 'recipe_dataset_index': 0, 'institute': ['IPSL'], 'activity': 'CMIP', 'alias': 'CMIP6_IPSL-CM6A-LR', 'original_short_name': 'tas', 'standard_name': 'air_temperature', 'long_name': 'Near-Surface Air Temperature', 'units': 'K', 'modeling_realm': ['atmos'], 'frequency': 'mon', 'filename': '/home/users/valeriu/esmvaltool_var_test/esmvaltool_output/recipe_validation_CMIP6_20211012_121208/preproc/validation_with_ERA-Interim/tas/CMIP6_IPSL-CM6A-LR_Amon_historical_r1i1p1f1_tas_1900-2000.nc'}, check your recipe entry
There was a problem hiding this comment.
we can do that later, in a new PR if you think the current behavior is enough for now though 👍
There was a problem hiding this comment.
Missing data is a RecipeError
There was a problem hiding this comment.
I changed that recently, look at the code above. You may need to do a git pull if you still get a KeyError.
There was a problem hiding this comment.
You're right we could catch this error before even looking for data, could you open a new issue suggesting that as an improvement?
There was a problem hiding this comment.
I changed that recently, look at the code above. You may need to do a git pull if you still get a KeyError.
ah awesome, cheers!
You're right we could catch this error before
Shall dos! But isn't that a trivial fix by adding grid to the required_keys list in _recipe.py (if we want to promote it to being so important, that is)
There was a problem hiding this comment.
Feel free to put that information in the issue.
There was a problem hiding this comment.
Yes, but we could also be a bit smarter and extract required_keys from the project DRS
Description
Closes #1134
Link to documentation:
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: