How to design the way the metadata file is generated for each MIP experiment?

[treerink]

In fact the question is how the metadata file is generated and how it is used for each MIP experiment. It seems that quite some fields need to be manually filled out or at least checked for each MIP, some will be always the same, others could be generated from the CV file.

The meaning of the fields in this metadata-template.json is described in this google doc by Klaus.

[klauswyser]

The Google doc certainly needs an update, I can have a look at it next week. What I wonder is whether or not is is sufficient to have the Google doc (or similar) with instructions and the leave it to every users of the ece2cmor3 tool to make sure that everything is filled in correctly? Or could we have a utility that would create the metadata template from user provided input (MIP, experiment, etc)? I have no idea how complicated such a utility would be.

[treerink]

We are thinking of generating the meta file (or several meta files for each experiment, see remark below), by the generate-ec-earth-namelists.sh script and providing them another subdir in the each namelist directory. But they will need manual adjustment, however if this will be done under the EC-Earth repository (as soon its archived there, as we plan I think), then of course this manual adjustment has to be done only once per meta file. It will be possible, at time of auto generation by the generate-ec-earth-namelists.sh script to extract a part of the meta data (will be some work), but might be worth the effort because the large number of experiments in total.

Probably we need several meta files for each experiment (at the moment Gijs uses one for NEMO and one for IFS, because of the grid labeling in the file names), for IFS, NEMO and maybe different ones for LPJ-GUESS and TM5 as well.

Anyway an update of the description file will be certainly handy before starting this.

[treerink]

About the contact email address: philippe.le.sager@knmi.nl are we going to specify just one central mail address for all experiments? Is it possible to have a mail address like cmip6-results@ec-earth.org or otherwise just take a gmail for it. Or do we provide personal mail addresses for each experiment (this might be a bit chaotic and soon outdated, but the the advantage is that it is quite direct). Or both?

A web address is maybe also possible (if the checks allow).

[treerink]

A CMIP6 CMIP piControl template of an metadat.json file is added in:

 resources/metadata-templates/cmip6-CMIP-piControl-metadata-template.json

Some inside-quotes-comments concern fields that we probably don't have to specify in the metadata file because they will be anyhow overwritten by cmor using the CV data. This file needs some more discussion and testing and if the basic template for a CMIP6 experiment has its final form, we can think of generating other MIP experiment metadata templates.

[treerink]

The resources/metadata-templates/cmip6-CMIP-piControl-metadata-template.json file can now be used to cmorize CMIP6 piControl results. That means ece2cmor3 won't stop with an error message. The correctness of the output should be validated once more.

[klauswyser]

I went through the resources/metadata-templates/cmip6-CMIP-piControl-metadata-template.json file and cleaned it even more because some attributes don't need to be set manually (e.g. data_specs_version).
Here is my version of metadata-piControl.json.txt (sorry, had to rename it, json format nor suported)

Notes:

1. The source attribute is set automatically from source_id, nevertheless the source attribute cannot be ommitted from the matadata file (yet its value doesn't matter)
2. The contact address is cmip6-data@ec-earth.org (not cmip6-results)
3. The metadata file description has been updated and follows the structure of the new template

[treerink]

We would like to remove the resources/metadata-template.json file, this needs revision of the nosetests

[treerink]

Next task is to be able to remove the following lines

    "grid":                         "T511-ORCA025",
    "grid_label":                   "gr",

from the meta data file:

resources/metadata-templates/cmip6-CMIP-piControl-metadata-template.json

Therefore ece2cmor3 will write this grid info to the netcdf files directly, and it will be only specific to the model component in charge. In this way the correct extension for the TM5 and LPJ-GUESS grid info can also easily be added.

[treerink]

Hi Gijs,

It is still not possible to remove the resources/metadata-template.json file #214 (comment) . The nosetests fail in that case:

! In function: cmor_open_inpathFile
! called from: cmor_dataset_json
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Could not find file: /home/thomas/cmorize/ece2cmor3/ece2cmor3/resources/metadata-template.json
!
!!!!!!!!!!!!!!!!!!!!!!!!!

[goord]

@treerink can we either close this issue or remove the release tag since it won't be resolved before the release? Shall we for now just point people to the documentation that Klaus wrote, and give support either here, on the EC-Earth portal or via email on adapting the template to the MIP/experiment?

[treerink]

As it is not possible to add decent comment in the metadata file itself, it is maybe good to add a README in the resources/metadata-templates/ dir. It can point to Klaus' documents and also emphasis the main points to think about when modifying the metadata template. And also stress the archiving policy for other metadata template files. This README can then also be referenced from elsewhere, e.g. from the README.md

[treerink]

If ece2cmor3 determines the grid info automatically, it can be omitted in the metadata template json file which no longer makes your template model component dependent. Several of the metadata fields actually can be obtained from the CMIP6_CV.json file including the grid info (or we hard code that grid info). We target to make this happen soon.

[treerink]

The script modify-metadata-template.sh is to large extent able to generate for each MIP experiment a dedicated metadata template. It will be added to genecec in order to produce all metadata templates (variants) in the control output files directories. Of course the user always has to inspect the metadata file, especially for the member info and for the "parent" info. The latter has been not addressed so far in the modify-metadata-template.sh script, I don't know yet if this is fully controlled possible. Anyway I think the current MIP experiment dedicated templates might be quite well helpful.

[treerink]

Finally the parent info is also part of the adjustment in the modify-metadata-template.sh script, and the production of all those metadata template files per MIP experiment and per component (i.e. ifs, nemo, tm5 and/or lpjg) are added to genecec.

[treerink]

So done. Closing.

[ufladrich]

At least in

CMIP/EC-EARTH-Veg/cmip6-experiment-CMIP-piControl/ \
    metadata-cmip6-CMIP-piControl-EC-EARTH-Veg-nemo-template.json

it says

   "grid":                         "ORCA1L75",
    "grid_label":                   "gn",
    "nominal_resolution":           "50 km",

but I think ORCA1 provides a nominal resolution of 100 km.

[uwefladrich]

@klauswyser What's your view on this? 50 km is not what is says in your google doc on metadata for EC-Earth, but @treerink tells me that it is what is should be.

[klauswyser]

Well, 50 km is wrong and needs to be changed!

Computing the average maximal length of a gridcell (with land points masked!) according to Appendix 2 of the document that describes the CMIP6 metadata yields 120.6 km which is larger than 72 km and therefore the nominal resolution should be 100 km.

BTW: here is the cdo command that computes the average maximum gridcell length in case you don't believe me:

cdo output -div -fldsum -expr,'x=e1t*e2t*tmaskutil*sqrt(sqr(e1t)+sqr(e2t))' mesh_mask.nc -fldsum -expr,'y=e1t*e2t*tmaskutil' mesh_mask.nc

[treerink]

I always believe you Klaus, I see also overhere that ORCA1 is always classified with a 100 km nominal resolution, so I will adjust.

Lets check everything:

  ECE CONF NAME        IFS RES    NEMO RES      TM5 RES                                  LPJG RES   PISCES RES  PISM RES   source_type AOGCM  source_type AGCM
  EC-Earth3            T255L91    ORCA1L75      none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-HR         T511L91    ORCA025L75    none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-LR         T159L91    ORCA1L75      none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-CC         T255L91    ORCA1L75      native regular 2x3 degree latxlon grid    T255L91    ORCA1L75    none       AOGCM BGC          AGCM BGC       
  EC-Earth3-GrIS       T255L91    ORCA1L75      none                                      none       none        5 x 5 km   AOGCM ISM          AGCM ISM       
  EC-Earth3-AerChem    T255L91    ORCA1L75      native regular 2x3 degree latxlon grid    none       none        none       AOGCM AER CHEM     AGCM AER CHEM  
  EC-Earth3-Veg        T255L91    ORCA1L75      none                                      T255L91    none        none       AOGCM              AGCM           
  EC-Earth3-Veg-LR     T159L91    ORCA1L75      none                                      T159L91    none        none       AOGCM              AGCM           

  EC-Earth3            100 km     100 km        none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-HR          50 km      25 km        none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-LR         100 km     100 km        none                                      none       none        none       AOGCM              AGCM           
  EC-Earth3-CC         100 km     100 km        250 km                                    100 km     100 km      none       AOGCM BGC          AGCM BGC       
  EC-Earth3-GrIS       100 km     100 km        none                                      none       none        5 km       AOGCM ISM          AGCM ISM       
  EC-Earth3-AerChem    100 km     100 km        250 km                                    none       none        none       AOGCM AER CHEM     AGCM AER CHEM  
  EC-Earth3-Veg        100 km     100 km        none                                      100 km     none        none       AOGCM              AGCM           
  EC-Earth3-Veg-LR     100 km     100 km        none                                      100 km     none        none       AOGCM              AGCM           

Any other errors? Please also check the source type (last columns), do we agree?

[treerink]

An update with genecec with this correction has been committed in the EC-Earth3 r6874-control-output-files branch, see issue 659 ECE portal.

Closing.