Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cam6_3_148: Provide RRTMGP as a radiation parameterization #909

Merged
merged 70 commits into from
Feb 21, 2024

Conversation

brian-eaton
Copy link
Collaborator

@brian-eaton brian-eaton commented Oct 25, 2023

Issue #255 - Provide RRTMGP as a radiation parameterization

resolves #255

Misc:
. The use case file 1850_cam5.xml is returned to the source code. This is to facilitate running the F1850 compset with CAM5 physics as discussed in issue #393.

@jedwards4b
Copy link

@cacraigucar there are a number of ways to limit the checked out repository in git, specifically

 --depth <depth>
           Create a shallow clone with a history truncated to the specified number of commits. Implies --single-branch unless
           --no-single-branch is given to fetch the histories near the tips of all branches. If you want to clone submodules
           shallowly, also pass --shallow-submodules.

git submodule add also has a --depth option.

However I don't think that there is an easy way to apply them with manage_externals.

@cacraigucar
Copy link
Collaborator

@cacraigucar there are a number of ways to limit the checked out repository in git, specifically

 --depth <depth>
           Create a shallow clone with a history truncated to the specified number of commits. Implies --single-branch unless
           --no-single-branch is given to fetch the histories near the tips of all branches. If you want to clone submodules
           shallowly, also pass --shallow-submodules.

git submodule add also has a --depth option.

However I don't think that there is an easy way to apply them with manage_externals.

Doing a clone by hand with --depth 1 doesn't appear to impact the size at all. The .git/objects/pack directory is still the same size. Any other suggestions?

@jedwards4b
Copy link

Actually you can bring these in as submodules using the --depth option as suggested and then use the manage_externals interface to submodules:

[rrtmgp]
protocol = git
from_submodule = True
required = True

@jedwards4b
Copy link

Perhaps you made an error?

git clone -b v1.7 --depth 1 https://github.com/earth-system-radiation/rte-rrtmgp.git
Cloning into 'rte-rrtmgp'...
remote: Enumerating objects: 126, done.
remote: Counting objects: 100% (126/126), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 126 (delta 10), reused 66 (delta 6), pack-reused 0
Receiving objects: 100% (126/126), 204.39 KiB | 1.70 MiB/s, done.
Resolving deltas: 100% (10/10), done.
Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

 git switch -c <new-branch-name>

Or undo this operation with:

 git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp
:) du -sk rte-rrtmgp/
3186	rte-rrtmgp/

git clone -b v1.7 https://github.com/earth-system-radiation/rte-rrtmgp.git rte-rrtmgp.full
Cloning into 'rte-rrtmgp.full'...
remote: Enumerating objects: 7537, done.
remote: Counting objects: 100% (2860/2860), done.
remote: Compressing objects: 100% (873/873), done.
remote: Total 7537 (delta 2082), reused 2644 (delta 1920), pack-reused 4677
Receiving objects: 100% (7537/7537), 118.07 MiB | 18.80 MiB/s, done.
Resolving deltas: 100% (5213/5213), done.
Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp
:) du -sk rte-rrtmgp.full/
242583 rte-rrtmgp.full/


@cacraigucar
Copy link
Collaborator

Perhaps you made an error?

git clone -b v1.7 --depth 1 https://github.com/earth-system-radiation/rte-rrtmgp.git
Cloning into 'rte-rrtmgp'...
remote: Enumerating objects: 126, done.
remote: Counting objects: 100% (126/126), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 126 (delta 10), reused 66 (delta 6), pack-reused 0
Receiving objects: 100% (126/126), 204.39 KiB | 1.70 MiB/s, done.
Resolving deltas: 100% (10/10), done.
Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

 git switch -c <new-branch-name>

Or undo this operation with:

 git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp
:) du -sk rte-rrtmgp/
3186	rte-rrtmgp/

git clone -b v1.7 https://github.com/earth-system-radiation/rte-rrtmgp.git rte-rrtmgp.full Cloning into 'rte-rrtmgp.full'... remote: Enumerating objects: 7537, done. remote: Counting objects: 100% (2860/2860), done. remote: Compressing objects: 100% (873/873), done. remote: Total 7537 (delta 2082), reused 2644 (delta 1920), pack-reused 4677 Receiving objects: 100% (7537/7537), 118.07 MiB | 18.80 MiB/s, done. Resolving deltas: 100% (5213/5213), done. Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp :) du -sk rte-rrtmgp.full/ 242583 rte-rrtmgp.full/


Yup - I must have done something wrong, as my size is dramtically reduced.

What exactly do I put in the Externals.cfg file?

Perhaps you made an error?

git clone -b v1.7 --depth 1 https://github.com/earth-system-radiation/rte-rrtmgp.git
Cloning into 'rte-rrtmgp'...
remote: Enumerating objects: 126, done.
remote: Counting objects: 100% (126/126), done.
remote: Compressing objects: 100% (114/114), done.
remote: Total 126 (delta 10), reused 66 (delta 6), pack-reused 0
Receiving objects: 100% (126/126), 204.39 KiB | 1.70 MiB/s, done.
Resolving deltas: 100% (10/10), done.
Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

 git switch -c <new-branch-name>

Or undo this operation with:

 git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp
:) du -sk rte-rrtmgp/
3186	rte-rrtmgp/

git clone -b v1.7 https://github.com/earth-system-radiation/rte-rrtmgp.git rte-rrtmgp.full Cloning into 'rte-rrtmgp.full'... remote: Enumerating objects: 7537, done. remote: Counting objects: 100% (2860/2860), done. remote: Compressing objects: 100% (873/873), done. remote: Total 7537 (delta 2082), reused 2644 (delta 1920), pack-reused 4677 Receiving objects: 100% (7537/7537), 118.07 MiB | 18.80 MiB/s, done. Resolving deltas: 100% (5213/5213), done. Note: switching to '4d8c5df4c63434aaab854afd1b02f5986d41dfb3'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

casper-login1: ~/tmp :) du -sk rte-rrtmgp.full/ 242583 rte-rrtmgp.full/


Yup, I must have done something wrong as I now see the size I need. Sorry about that!

What exactly do we need to put into our workflow and/or manage_externals? RRTMGP in this PR is a part of Externals_CAM.cfg. When I add "from_submodule = True" into the rrtmgp sections, I get an error from manage_externals that I have over specified with both from_submodule and tag. I also don't know where I'd put in the --depth 1. I'm trying to find a solution that a user cloning CAM and running manage_externals will get the smaller sized checkout. Is this possible?

@jedwards4b
Copy link

You want to bring rrtmgp into cam as a submodule
git submodule add --depth=1 -b v1.7 https://github.com/earth-system-radiation/rte-rrtmgp.git src/physics/rrtmgp/ext
and add the code exactly as I have it above to Externals_CAM.cfg
You won't be able to add a tag to the Externals_CAM.cfg file.

@cacraigucar
Copy link
Collaborator

Thanks @jedwards4b . I think I finally understand (I've never dealt with submodules before).

@brian-eaton - Can you add the two rrtmgp checkouts as submodules (using the command Jim provided above with the --depth 1)? Then you'll need to modify the Externals_CAM.cfg file for these two externals. This should reduce the RRTMGP size tremendously. I've done this by hand and it appears to work for me, so let me know if you have any questions.

@jedwards4b
Copy link

Just so you are aware for developers - you can go to the submodule directory and run git fetch origin to get the entire repository locally if you want it.

src/physics/cam/cloud_rad_props.F90 Outdated Show resolved Hide resolved
src/physics/cam/cloud_rad_props.F90 Outdated Show resolved Hide resolved
src/physics/cam/cloud_rad_props.F90 Outdated Show resolved Hide resolved
src/physics/cam/cloud_rad_props.F90 Outdated Show resolved Hide resolved
src/physics/rrtmgp/radiation.F90 Show resolved Hide resolved
src/physics/rrtmgp/radiation.F90 Outdated Show resolved Hide resolved
src/physics/rrtmgp/radiation.F90 Outdated Show resolved Hide resolved
@@ -1221,7 +1242,7 @@ if ($carma eq 'bc_strat') {
}
}

if ($rrtmg) {
if ($rad_pkg eq 'rrtmg') {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just leaving this comment un-resolved so that we remember to make a Github issue about CARMA not working with RRTMGP after this PR has been merged.

Copy link
Collaborator

@nusbaume nusbaume left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for responding to all of my comments! This PR looks good to me now. I did find one last set of typos, and one optional suggestion, but I don't believe it needs another review from me.

src/physics/rrtmgp/radiation.F90 Show resolved Hide resolved
Copy link
Collaborator

@cacraigucar cacraigucar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submitting my comments from my short review (lots still to go).

bld/configure Outdated Show resolved Hide resolved
Comment on lines +89 to +94
[rte-rrtmgp]
local_path = src/physics/rrtmgp/ext
protocol = git
repo_url = https://github.com/earth-system-radiation/rte-rrtmgp.git
tag = v1.7
required = True
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change these once rrtmgp is brought in as a submodule

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per comments below (thanks @jedwards4b and @nusbaume) will leave this alone for now. Will revisit when we bring in git-fleximod.

cime_config/testdefs/testlist_cam.xml Show resolved Hide resolved
cime_config/testdefs/testlist_cam.xml Outdated Show resolved Hide resolved
Copy link
Collaborator

@cacraigucar cacraigucar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next round of comments - still need to look at a few more routines

src/physics/cam/cloud_rad_props.F90 Outdated Show resolved Hide resolved
src/physics/rrtmgp/mcica_subcol_gen.F90 Show resolved Hide resolved
src/physics/rrtmgp/radconstants.F90 Outdated Show resolved Hide resolved
src/physics/rrtmgp/radconstants.F90 Show resolved Hide resolved
@brian-eaton
Copy link
Collaborator Author

You want to bring rrtmgp into cam as a submodule git submodule add --depth=1 -b v1.7 https://github.com/earth-system-radiation/rte-rrtmgp.git src/physics/rrtmgp/ext and add the code exactly as I have it above to Externals_CAM.cfg You won't be able to add a tag to the Externals_CAM.cfg file.

@jedwards4b, I'm trying to add the RRTMGP source as a submodule following your instructions, but am getting an error. Here's what happens, starting from a clean clone of my CAM fork on branch rrtmgp:

% cd /work/src
% git clone -b rrtmgp https://github.com/brian-eaton/CAM cam-rrtmgp
remote: Enumerating objects: 34903, done.
remote: Counting objects: 100% (1191/1191), done.
remote: Compressing objects: 100% (565/565), done.
remote: Total 34903 (delta 724), reused 992 (delta 601), pack-reused 33712
Receiving objects: 100% (34903/34903), 39.37 MiB | 22.91 MiB/s, done.
Resolving deltas: 100% (25327/25327), done.

% cd cam-rrtmgp
% git status
On branch rrtmgp
Your branch is up to date with 'origin/rrtmgp'.

nothing to commit, working tree clean

% git submodule add --depth=1 -b v1.7 \
  https://github.com/earth-system-radiation/rte-rrtmgp.git src/physics/rrtmgp/ext
The following paths are ignored by one of your .gitignore files:
src/physics/rrtmgp/ext
hint: Use -f if you really want to add them.
hint: Turn this message off by running
hint: "git config advice.addIgnoredFile false"

So I added the -f option:

% git submodule add --depth=1 -b v1.7 -f \
  https://github.com/earth-system-radiation/rte-rrtmgp.git src/physics/rrtmgp/ext
Cloning into '/work/src/cam-rrtmgp/src/physics/rrtmgp/ext'...
remote: Enumerating objects: 129, done.
remote: Counting objects: 100% (129/129), done.
remote: Compressing objects: 100% (117/117), done.
remote: Total 129 (delta 11), reused 58 (delta 5), pack-reused 0
Receiving objects: 100% (129/129), 206.66 KiB | 2.46 MiB/s, done.
Resolving deltas: 100% (11/11), done.
fatal: 'origin/v1.7' is not a commit and a branch 'v1.7' cannot be created from it
fatal: unable to checkout submodule 'src/physics/rrtmgp/ext'

I don't understand this error. v1.7 is a tag on branch main in the rte-rrtmgp repo. Any ideas?

@nusbaume
Copy link
Collaborator

Hi @brian-eaton @jedwards4b @cacraigucar I have been working on this today, and I have reached a point where I believe there is a bug in manage_externals which prevents submodules from working properly. When I add the two RRTMGP repos as submodules (which I know is working because I can use commands like git submodule update to get the relevant files but with a much smaller disk footprint), but then try to use checkout_externals with the from_submodules flag I get the following error:

{   'externals': '',
    'from_submodule': True,
    'repo': {   'branch': '',
                'hash': '',
                'protocol': 'git',
                'repo_url': '',
                'sparse': '',
                'tag': ''},
    'required': True}
ERROR:root:No parent submodule for "rte-rrtmgp"

The problem is that the .gitmodules file definitely exists in the top-level directory, and definitely contains a submodule named rte-rrtmgp. Quickly looking through the code it appears that the issue could be that there is a parent_repo variable that doesn't seem to be set anywhere, although I could certainly be wrong on that front.

If I am wrong here and someone else can get it to work properly then definitely ignore this comment post. However if everyone else runs into the same problem then at this point I might vote to just leave the checkout as-is, and then once we bring in git-fleximod we'll implement a single depth checkout to reduce the large disk space overhead. To me that seems like a better use of everyone's time then trying to fix software (manage_externals) which is going away soon. Thoughts?

@jedwards4b
Copy link

@nusbaume is correct, this is a manage_extenals bug - if you checkout cesm so that cam is not the toplevel directory the submodule checkout will work correctly.

@brian-eaton Reverse the --depth and -b options:

git submodule add -b v1.7 --depth=1 -f https://github.com/earth-system-radiation/rte-rrtmgp.git src/physics/rrtmgp/ext
Adding existing repo at 'src/physics/rrtmgp/ext' to the index

Copy link
Collaborator

@cacraigucar cacraigucar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final set of review comments.

Kudos to whoever wrote the excellent code comments (especially in the rrtmgp_inputs.F90 module). They really helped with reviewing the code!

src/physics/rrtmgp/radconstants.F90 Show resolved Hide resolved
src/physics/rrtmgp/rrtmgp_inputs.F90 Show resolved Hide resolved
Comment on lines +72 to +76
! Mapping from RRTMG shortwave bands to RRTMGP. Currently needed to continue using
! the SW optics datasets from RRTMG (even thought there is a slight mismatch in the
! band boundaries of the 2 bands that overlap with the LW bands).
integer, parameter, dimension(14) :: rrtmg_to_rrtmgp_swbands = &
[ 14, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a check to make sure that the bands in RRTMGP have not changed position (assuming this is a list of index locations).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike the RRTMG code, the RRTMGP interface code doesn't make any assumption about the spectral band boundaries or their order. That data is read from input datasets. What the RRTMGP code is assuming is the band order in the RRTMG optics datasets. The reason this mapping is hard-coded is that the optics datasets don't contain the metadata that they should contain to describe the spectral bands. Furthermore a couple of the bands used by RRTMGP don't have the same bounds that the RRTMG versions have. The bottom line is that the optics datasets used by RRTMGP should be updated to match the RRTMGP band boundaries. But this hack enables continued use of the RRTMG datasets until the scientists decide the approximation involved is no longer acceptable.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this comment "unresolved" as it will make it easier to find this explanation.

Comment on lines +151 to +152
! Add extra layer values if needed.
if (nlay == pverp) then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this work with all variations of CAM current and future since model tops can change? I would think a better check would be to retrieve the pressure at the top and make the if check based off of that

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nlay depends on the number of model layers that are below the RRTMGP top of 1 Pa. It is set in subroutine radiation_init making use of the reference pressures at the model layer interfaces. The condition nlay == pverp tells you that the radiation is using 1 more than the pver layers that CAM has, hence the addition of the "extra layer". Dealing with the fact that RRTMGP usually uses a different number of layers than CAM is the most subtle aspect of the interface code. But this is also true for RRTMG, so we were able to leverage a lot of existing code to deal with this issue.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. I am leaving this unresolved so we can easily see your explanation about this important aspect of the code

src/physics/rrtmgp/rrtmgp_inputs.F90 Show resolved Hide resolved
src/physics/rrtmgp/rrtmgp_inputs.F90 Show resolved Hide resolved
Copy link
Collaborator

@cacraigucar cacraigucar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job everyone on this PR!!!

Comment on lines +151 to +152
! Add extra layer values if needed.
if (nlay == pverp) then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. I am leaving this unresolved so we can easily see your explanation about this important aspect of the code

Comment on lines +72 to +76
! Mapping from RRTMG shortwave bands to RRTMGP. Currently needed to continue using
! the SW optics datasets from RRTMG (even thought there is a slight mismatch in the
! band boundaries of the 2 bands that overlap with the LW bands).
integer, parameter, dimension(14) :: rrtmg_to_rrtmgp_swbands = &
[ 14, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this comment "unresolved" as it will make it easier to find this explanation.

src/physics/rrtmgp/mcica_subcol_gen.F90 Show resolved Hide resolved
Comment on lines 202 to 209
<test compset="QPC6" grid="ne5pg3_ne5pg3_mg37" name="ERP_D_Ln9" testmods="cam/outfrq9s_rrtmgp">
<machines>
<machine name="izumi" compiler="gnu" category="aux_cam"/>
</machines>
<options>
<option name="wallclock">00:10:00</option>
</options>
</test>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to keep this in mind if more tests start to use RRTMGP (or if it ever becomes default).

@brian-eaton brian-eaton merged commit 660e3fd into ESCOMP:cam_development Feb 21, 2024
cacraigucar added a commit to cacraigucar/CAM that referenced this pull request Feb 21, 2024
Merge pull request ESCOMP#909 from brian-eaton/rrtmgp
@cacraigucar cacraigucar changed the title Provide RRTMGP as a radiation parameterization cam6_3_148: Provide RRTMGP as a radiation parameterization Feb 21, 2024
gdicker1 added a commit to EarthWorksOrg/CAM that referenced this pull request Mar 5, 2024
ESCOMP/CAM PR ESCOMP#909 from brian-eaton/rrtmgp. Provide RRTMGP as a
radiation parameterization.
gdicker1 added a commit to EarthWorksOrg/CAM that referenced this pull request Mar 5, 2024
ESCOMP/CAM PR ESCOMP#909 from brian-eaton/rrtmgp. Provide RRTMGP as a
radiation parameterization.
gold2718 pushed a commit to gold2718/CAM that referenced this pull request May 2, 2024
Merge pull request ESCOMP#909 from brian-eaton/rrtmgp

Provide RRTMGP as a radiation parameterization

ESCOMP commit: 660e3fd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFB bit for bit tag CoupledEval3
Projects
Status: Tag
Development

Successfully merging this pull request may close these issues.

6 participants