Build a new live-CMORiser for ACCESS-ESM raw data in ESMValTool

[rhaegar325]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is.

Would you be able to help out?
Would you have the time and skills to implement the solution yourself?

Hi, ESMValGroup:
I am from ACCESS-NRI, recently we trying to build a live-CMORiser in ESMValTool for ACCESS-ESM raw data, our purpose is to make it able to live-CMORise any variable in ACCESS-ESM dataset. Now we already have a useful tool named APP4(Access post-processor), it can CMORise every variable in ACCESS-ESM, so the simplest way is bind APP4 as a CMORiser or a preprocessor in ESMValTool. Since I am new to ESMValTool, I raise a issue here to see if anyone have any related experience or could provide any suggestions about how to build a CMORiser for ESMValTool(e.g. interface, related docs), that would be super helpful.

[schlunma]

Hi @rhaegar325, welcome to the community! It's great to hear that you want to use ESMValTool to process ACCESS-ESM data 🚀.

What you describe here has been already implemented for 5 other models. We usually refer to it as "on-the-fly CMORization" or "CMOR-like reformatting". Implementing support for a new model usually involves 3 main steps:

1. Expand config-developer.yml with the input file and directory structure used by ACCESS-ESM (example for the ICON model).
2. Add an extra facets file for ACCESS-ESM that contains mappings from CMOR variables to ACCESS-ESM-specific entries (usually elements of the paths from 1.), see example for ICON model.
3. Add a Fix for ACCESS-ESM that actually contains the code to make the model data CMOR-compliant. Usually this contains a lot of metadata changes. Here is an example for the ICON model (complicated) and here a much simpler one for CESM2. I really recommend to check out the NativeDatasetFix base class used in both cases.

All of this is explained in more detail here. An overview of all currently supported models is given here. If you want to know more about our thoughts and ideas about the design of this framework, check out this paper.

Regarding the APP4 tool that you mentioned: if this is a Python package that would be super helpful! You could then simply import and use this in the fix function and probably avoid a lot of code duplication.

Let me know if you have any further questions, I am happy to help! Please also feel free to open a draft pull request at any development stage, as this usually makes it much easier for us to support you! Thanks!

[rhaegar325]

Hi, @schlunma:

Appreciate a lot for your reply, that would be really helpful. I would follow your instruction to start a new branch to have a try, and I would like to share with you if I make any progress or meet any problem through the process.

keep in touch.