on-the-fly cmoriser for ACCESS native data #2430

rhaegar325 · 2024-05-21T06:56:05Z

Description

This pull request is to add the first version of on-the-fly cmoriser for ACCESS native data.
in this version we support only two variables(tas, pr), and will try to develop a more general version in next version.

Closes ESMValGroup/ESMValTool#3431

Link to documentation:

Before you get started

☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🧪 The new functionality is relevant and scientifically sound
🛠 This pull request has a descriptive title and labels
🛠 Code is written according to the code quality guidelines
🧪 and 🛠 Documentation is available
🛠 Unit tests have been added
🛠 Changes are backward compatible
🛠 Any changed dependencies have been added or removed correctly
🛠 The list of authors is up to date
🛠 All checks below this pull request were successful

To help with the number pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

rhaegar325 · 2024-05-21T07:04:17Z

@rbeucher our on-the-fly cmoriser pr is there, fixing issues to pass those tests

environment.yml

rbeucher · 2024-05-21T21:54:49Z

Thanks Rhaegar. I think we need to update the documentation too. I will have a go.

codecov · 2024-05-22T04:21:39Z

Codecov Report

Attention: Patch coverage is 36.53846% with 33 lines in your changes are missing coverage. Please review.

Project coverage is 93.85%. Comparing base (f72a873) to head (0e864b6).
Report is 39 commits behind head on main.

Files	Patch %	Lines
esmvalcore/cmor/_fixes/access/access_esm.py	36.53%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2430      +/-   ##
==========================================
- Coverage   94.07%   93.85%   -0.23%     
==========================================
  Files         241      242       +1     
  Lines       13422    13474      +52     
==========================================
+ Hits        12627    12646      +19     
- Misses        795      828      +33

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

remove csv and os

codecov-commenter · 2024-05-24T02:11:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.64%. Comparing base (1da5bbc) to head (987bdd9).
Report is 13 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2430      +/-   ##
==========================================
+ Coverage   94.61%   94.64%   +0.02%     
==========================================
  Files         246      248       +2     
  Lines       14048    14105      +57     
==========================================
+ Hits        13292    13349      +57     
  Misses        756      756

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Small change to ACCESS Live CMORiser documentation

rbeucher · 2024-05-24T04:40:37Z

Hi @rhaegar325 ,

See the PR here #1678 for CESM2
I think you need to add some tests and also make sure that the general structure, including the mapping information, is in the correct location.

rbeucher

Hi Rhaegar.

It looks cleaner. However, specific variable fixes should be defined in specific classes. AllVar is supposed to be general and shouldn't have conditional sections based on the variable. See @schlunma comments.

esmvalcore/cmor/_fixes/access/access_esm.py

esmvalcore/cmor/_fixes/access/access_esm1_5.py

esmvalcore/cmor/_fixes/access/_base_fix.py

rbeucher

Just a few comments. I think it is good to go. I am not sure about the sub_dataset facet name... @flicj191 any ideas?

rbeucher · 2024-07-08T00:43:20Z

doc/quickstart/find_data.rst

+``sub_dataset``      Part of the ACCESS-ESM raw dataset     No default
+                     root, need to specify if you want to
+                     use the cmoriser                                                                       


Is it part of the root ? I don't think so. careful to not create confusion with the rootpath in the config file.

rbeucher · 2024-07-08T00:43:55Z

esmvalcore/cmor/_fixes/access/access_esm1_5.py

+class Rlus(AccessFix):
+    """Fixes for Rlus."""
+
+    def fix_rlus_data(self, cubes):


rename to fix_data

rbeucher · 2024-07-08T00:44:12Z

esmvalcore/cmor/_fixes/access/access_esm1_5.py

+class Rsus(AccessFix):
+    """Fixes for Rsus."""
+
+    def fix_rsus_data(self, cubes):


rename to fix_data

esmvalcore/cmor/_fixes/access/_base_fix.py

rbeucher · 2024-07-08T13:18:40Z

You need to fix the tests @rhaegar325

Just bypassing this

schlunma · 2024-07-09T08:18:57Z

Hi @rbeucher, sorry, since there were so many commits on this lately I was waiting for a comment saying that it's ready for a final check. I would have loved to take one more look at this.

Before dismissing someone's review, please ask them and give them some time. If you don't get an answer after that, it's of course fine to dismiss the review. As I said, I wasn't aware that this is ready for another review. Not a big problem of course, just wanted to make you aware of that!

Thanks @rhaegar325 and everyone for all the work on this, it's great to have support for native output of one more model 🚀

rbeucher · 2024-07-09T08:22:57Z

Sorry @schlunma. I just had to make an executive decision on this one. We are preparing for a training event and we need to get things rolling. Thanks for reviewing this and sorry again.

Co-authored-by: Yousong Zeng <Yousong> Co-authored-by: Romain Beucher <romain.beucher@anu.edu.au> Co-authored-by: Manuel Schlund <32543114+schlunma@users.noreply.github.com>

Yousong Zeng added 6 commits March 28, 2024 14:15

branch for access live cmoriser

b774a36

update simple version for live cmoriser

5acf65c

fix issue with multi-model-statistics

e7277a0

comments added, ready for pull request

a0a05bc

add packages in environment.yml

a32a48f

change file name

efcea11

rbeucher reviewed May 21, 2024

View reviewed changes

environment.yml Outdated Show resolved Hide resolved

environment.yml Outdated Show resolved Hide resolved

Yousong Zeng added 2 commits May 22, 2024 14:07

reformat

9c0c9b8

change class name to CapWords convention

f98406d

rhaegar325 and others added 10 commits May 22, 2024 14:24

Update environment.yml

2b5f8d5

remove csv and os

Update access_esm.py

8043643

reformat

a86e1d9

reformat

c8bc262

summary line and description

f389559

reformat

7815a07

reformat

6db0088

reformat

4018368

reformat

0e864b6

update documentation

200e2d6

Yousong Zeng and others added 4 commits May 24, 2024 14:23

update documentation

803c544

update documentation

fbfefdc

update documentation

88047fc

Update find_data.rst

0cd3498

Small change to ACCESS Live CMORiser documentation

Yousong Zeng added 2 commits May 24, 2024 16:08

update config-developer.yml

7cd6cf8

restore some changes

7c91b6c

fix test

f429566

rbeucher requested changes Jul 4, 2024

View reviewed changes

esmvalcore/cmor/_fixes/access/access_esm.py Outdated Show resolved Hide resolved

esmvalcore/cmor/_fixes/access/access_esm1_5.py Outdated Show resolved Hide resolved

esmvalcore/cmor/_fixes/access/access_esm1_5.py Outdated Show resolved Hide resolved

rhaegar325 added 12 commits July 5, 2024 20:22

change class structure of cmoriser

d48dac9

change class structure of cmoriser

043b229

change class structure of cmoriser

ab151ab

change class structure of cmoriser

78e9a68

change class structure of cmoriser

0e97c34

change class structure of cmoriser

f033eea

change class structure of cmoriser

0f2ff15

change class structure of cmoriser

43279b0

fix test

bc69800

fix test

f750d5a

fix test

81be7d3

fix test

fdf9f56

rbeucher reviewed Jul 5, 2024

View reviewed changes

esmvalcore/cmor/_fixes/access/_base_fix.py Outdated Show resolved Hide resolved

fix class structure

18d31cc

rbeucher reviewed Jul 8, 2024

View reviewed changes

esmvalcore/cmor/_fixes/access/_base_fix.py Outdated Show resolved Hide resolved

rbeucher and others added 3 commits July 8, 2024 10:59

Move fix_height method to Tas class

7370733

Fix blank spaces

c5acc53

rename to fix_data

54caba7

fix function name

987bdd9

rbeucher approved these changes Jul 9, 2024

View reviewed changes

rbeucher merged commit 6b4e3fb into ESMValGroup:main Jul 9, 2024
3 of 4 checks passed

rbeucher mentioned this pull request Jul 9, 2024

Fix CFF file #2476

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

on-the-fly cmoriser for ACCESS native data #2430

on-the-fly cmoriser for ACCESS native data #2430

rhaegar325 commented May 21, 2024 •

edited

Loading

rhaegar325 commented May 21, 2024

rbeucher commented May 21, 2024

codecov bot commented May 22, 2024 •

edited

Loading

codecov-commenter commented May 24, 2024 •

edited by codecov bot

Loading

rbeucher commented May 24, 2024

rbeucher left a comment

rbeucher left a comment

rbeucher Jul 8, 2024

rbeucher Jul 8, 2024

rbeucher Jul 8, 2024

rbeucher commented Jul 8, 2024

schlunma commented Jul 9, 2024 •

edited

Loading

rbeucher commented Jul 9, 2024

on-the-fly cmoriser for ACCESS native data #2430

on-the-fly cmoriser for ACCESS native data #2430

Conversation

rhaegar325 commented May 21, 2024 • edited Loading

Description

Before you get started

Checklist

rhaegar325 commented May 21, 2024

rbeucher commented May 21, 2024

codecov bot commented May 22, 2024 • edited Loading

Codecov Report

codecov-commenter commented May 24, 2024 • edited by codecov bot Loading

Codecov Report

rbeucher commented May 24, 2024

rbeucher left a comment

Choose a reason for hiding this comment

rbeucher left a comment

Choose a reason for hiding this comment

rbeucher Jul 8, 2024

Choose a reason for hiding this comment

rbeucher Jul 8, 2024

Choose a reason for hiding this comment

rbeucher Jul 8, 2024

Choose a reason for hiding this comment

rbeucher commented Jul 8, 2024

schlunma commented Jul 9, 2024 • edited Loading

rbeucher commented Jul 9, 2024

rhaegar325 commented May 21, 2024 •

edited

Loading

codecov bot commented May 22, 2024 •

edited

Loading

codecov-commenter commented May 24, 2024 •

edited by codecov bot

Loading

schlunma commented Jul 9, 2024 •

edited

Loading