Stochastic physics clean and updates to CA

[pjpegion]

PR Checklist

• Ths PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• If new or updated input data is required by this PR, it is clearly stated in the text of the PR.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsiblity to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

Provide a detailed description of what this PR does. What bug does it fix, or what feature does it add? Is a change of answers expected from this PR? Are any library updates included in this PR (modulefiles etc.)?

The pull request includes a clean-up of the stochastic physics code and updates to the stand alone unit tests, which are now in the unit_tests sub-directory.
This PR also includes changes to the CA code for reproducibility for different process layouts and ca_global restart capability.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.
(Remember, issues must always be created before starting work on a PR branch!)

• fixes CA fails debug mode with GNU compiler #770 Clean up of stochastic physics #823 control_ca test is not reproducible with 2-threads #743
• fixes stochastic_physics/issues/ufs_public_release: update version number for CCPP, disable dynamic CCPP, cleanup work and bug fixes for regression tests #25 and stochastic_physics/issues/Fix restarts and add restart tests for ufs_public_release #33
  -fixes Add ocean stochastic physics NOAA-EMC/MOM6#52

Testing

How were these changes tested? What compilers / HPCs was it tested with? Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)

Changes were tests on hera using intel and gnu compiler. There are 2 new tests, control_ca_restart and datm_stochy_gefs. In addition, the control_ca test results have changed.

• Hera.gnu, baselines change with addition of two tests:

control_ca_debug
control_stochy_debug

• Hera.intel, baselines change with one new test and 3 tests with answer changes:

datm_cdeps_stochy_gefs (new test)
cpld_ca
control_ca
control_ca_debug

These tests have do_ca=.true. All other tests do not currently use CA.

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss_cray
• wcoss_dell_p3
• CI: https://github.com/ufs-community/ufs-weather-model/actions/runs/1291499680

Dependencies

If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).

Do PRs in upstream repositories need to be merged first?
If so add the "waiting for other repos" label and list the upstream PRs

• MOM6 PR 72
• waiting on code updates for CA_global NCAR/ccpp-physics#737
• waiting on updates associated with CA restart and PE decomposition bugs NOAA-EMC/fv3atm#396
• waiting on Stochastic physics cleanup NOAA-PSL/stochastic_physics#47

[pjpegion]

I ran the full GNU regression tests on hera, and the model is crashing in the control_flake and control_stochy tests, and I will update and I figure out what is going on.

[pjpegion]

Found the bug, there was an uninitialized array in spectral_transforms.F90. Gnu regression tests now pass on hera.

[jiandewang]

@pjpegion
https://github.com/pjpegion/ufs-weather-model/blob/new_stoch/.gitmodules#L30-L32
you are still pointing MOM6 to EMC repo, needs to use your MOM PR branch for ufs testing

[pjpegion]

@pjpegion
https://github.com/pjpegion/ufs-weather-model/blob/new_stoch/.gitmodules#L30-L32
you are still pointing MOM6 to EMC repo, needs to use your MOM PR branch for ufs testing

Sorry, I always forget about the gitmodules. fixed now

[jiandewang]

@pjpegion
https://github.com/pjpegion/ufs-weather-model/blob/new_stoch/.gitmodules#L32
you forgot to modify the branch name here

[pjpegion]

@pjpegion
https://github.com/pjpegion/ufs-weather-model/blob/new_stoch/.gitmodules#L32
you forgot to modify the branch name here

I don't think the brach name matters since the checkout should point to a specific has, but I updated it, along with the stochastic_physics branch name

[junwang-noaa]

MOM6 PR points to an issue, not a PR

[DeniseWorthen]

@MinsukJi-NOAA did CI run?

[junwang-noaa]

Yes, CI ran successfully last night.

…

On Thu, Sep 30, 2021 at 8:16 AM Denise Worthen ***@***.***> wrote: @MinsukJi-NOAA <https://github.com/MinsukJi-NOAA> did CI run? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#832 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI7D6TJIGUFD4XCNSBOJYPTUERIJ7ANCNFSM5EWRLEQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jiandewang]

MOM6 PR has been merged

[MinsukJi-NOAA]

Failed control_restart CI can be fixed with this change related to the utest script. I think we can implement it either here or in the next PR.

diff --git a/tests/utests/rst.sh b/tests/utests/rst.sh
index edd1dc9..5d6f2de 100644
--- a/tests/utests/rst.sh
+++ b/tests/utests/rst.sh
@@ -4,6 +4,7 @@ source $PATHRT/utests/std.sh
 # Set up date and time of restart files for restart run
 FHROT=12
 RESTART_FILE_PREFIX="${SYEAR}${SMONTH}${SDAY}.$(printf "%02d" $(( SHOUR + FHROT  )))0000"
+DEP_RUN=${TEST_NAME}
 
 if [[ $application == 'global' ]]; then
   OUTPUT_FH="3 -1"
@@ -18,7 +19,6 @@ elif [[ $application == 'cpld' ]]; then
   RESTART_FILE_SUFFIX_HRS="${SYEAR}-${SMONTH}-${SDAY}-$(printf "%02d" $(( ${FHROT} )))"
   RESTART_FILE_SUFFIX_SECS="${SYEAR}-${SMONTH}-${SDAY}-$(printf "%02d" $(( ${FHROT}*3600 )))"
   RESTART_N=$((FHMAX-$FHROT))
-  DEP_RUN=${TEST_NAME}
 fi
 
 WARM_START=.T.

[DeniseWorthen]

Do SHOUR and FHROT and FHMAX need to be variables ($SHOUR,$FHROT,$FHMAX)?

[MinsukJi-NOAA]

Do SHOUR and FHROT need to be variables ($SHOUR,$FHROT)?

Not when used like this $(( SHOUR + FHROT ))

[pjpegion]

Is the change only moving DEP_RUN outside the if statement?

[MinsukJi-NOAA]

Is the change only moving DEP_RUN outside the if statement?

Yes

[pjpegion]

I'm just waiting in the merge of fv3atm.

[junwang-noaa]

@pjpegion fv3 PR was merged

[DeniseWorthen]

All submodules look correct to me.

[climbfuji]

@lisa-bengtsson any last words of wisdom, or are you ok with this PR? Please approve if yes.

[lisa-bengtsson]

@lisa-bengtsson any last words of wisdom, or are you ok with this PR? Please approve if yes.

Thank you for all your hard work everyone. I will approve this PR.

[DeniseWorthen]

I second the hard work comment. We'll be able to close multiple issues w/ this PR.

[MinsukJi-NOAA]

@DeniseWorthen CI all passed

[pjpegion]

Can we close this issue?

[DeniseWorthen]

Yes---this is the PR. Is there an associated issue that needs closing?