on-the-fly cmoriser for ACCESS native data

[rhaegar325]

Description

This pull request is to add the first version of on-the-fly cmoriser for ACCESS native data.
in this version we support only two variables(tas, pr), and will try to develop a more general version in next version.

Closes ESMValGroup/ESMValTool#3431

Link to documentation:

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[rhaegar325]

@rbeucher our on-the-fly cmoriser pr is there, fixing issues to pass those tests

[rbeucher]

Thanks Rhaegar. I think we need to update the documentation too. I will have a go.

[codecov]

Codecov Report

Attention: Patch coverage is 36.53846% with 33 lines in your changes are missing coverage. Please review.

Project coverage is 93.85%. Comparing base (f72a873) to head (0e864b6).
Report is 39 commits behind head on main.

Files	Patch %	Lines
esmvalcore/cmor/_fixes/access/access_esm.py	36.53%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2430      +/-   ##
==========================================
- Coverage   94.07%   93.85%   -0.23%     
==========================================
  Files         241      242       +1     
  Lines       13422    13474      +52     
==========================================
+ Hits        12627    12646      +19     
- Misses        795      828      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 76.47059% with 8 lines in your changes missing coverage. Please review.

Project coverage is 94.57%. Comparing base (1da5bbc) to head (e160394).
Report is 3 commits behind head on main.

❗ Current head e160394 differs from pull request most recent head f7081ba

Please upload reports for the commit f7081ba to get more accurate results.

Files	Patch %	Lines
esmvalcore/cmor/_fixes/access/access_esm.py	76.47%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2430      +/-   ##
==========================================
- Coverage   94.61%   94.57%   -0.05%     
==========================================
  Files         246      247       +1     
  Lines       14048    14082      +34     
==========================================
+ Hits        13292    13318      +26     
- Misses        756      764       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rbeucher]

Hi @rhaegar325 ,

See the PR here #1678 for CESM2
I think you need to add some tests and also make sure that the general structure, including the mapping information, is in the correct location.

[rhaegar325]

Hi, @schlunma and @rbeucher,
I have made some change based on @schlunma 's suggestion, and I think it's ready now. Please review it at you convenience, much appreciate.

[rbeucher]

@flicj191 @schlunma , would you have time to have a look again?

[flicj191]

It's working for me. Added another suggestion, sorry I didn't see in my previous review.

[flicj191]

Suggested change

	output_file: '{sub_dataset}.{special_attr}'
	output_file: '{institute}_{sub_dataset}_{special_attr}_{short_name}'

Could edit in the config-developer file so that the output preprocessed files can have more info like other output files, including short_name and institute and separated by underscores.

[rhaegar325]

Hi, @schlunma @valeriupredoi:
I add some function in this pr to make it can cmoriser more variables, however, due to some variables need to be calculate by other variables in the raw data, I need to use eval() to calculate it because the way to calculate for a specific variable is stored in extra_facets, but it can't pass the Codacy Static Code Analysis. Can we accept that by any chance?

[bouweandela]

I recommend using a more readable format here, e.g.
fld_s02i207 - fld_s02i201 + fld_s03i332 - fld_s02i205. Instead of using eval (which leaves a lot of room for user mistakes and potential security issues), you could just write a regular expression to parse the string to get out the variables and operators, which you then apply.