Update WW3: Sync develop branch to dev/ufs-weather-model branch - 2023-jun-27

[MatthewMasarik-NOAA]

Description

Updates the WW3 component by syncing changes in the develop branch to the dev/ufs-weather-model branch.

Top of commit queue on: TBD

Input data additions/changes

• No changes are expected to input data.
• There will be new input data.
• Input data will be updated.

Anticipated changes to regression tests:

• No changes are expected to any regression test.
• Changes are expected to the following tests:

Subcomponents involved:

• AQM
• CDEPS
• CICE
• CMEPS
• CMakeModules
• FV3
• GOCART
• HYCOM
• MOM6
• NOAHMP
• WW3
• stochastic_physics
• none

Combined with PR's (If Applicable):

NA

Commit Queue Checklist:

• Link PR's from all sub-components involved
• Confirm reviews completed in sub-component PR's
• Add all appropriate labels to this PR.
• Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
• Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

• Depends on Update dev/ufs-weather-model branch with develop - 2023-jun-27 NOAA-EMC/WW3#1029
• Closes WW3: Update dev/ufs-weather-model branch with develop #1808

Testing Day Checklist:

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
• Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

• Tested on hera, intel/gnu:
  • Note: The logs below capture the successful completion of all RT's. In the first run, all RT's were successful in (i.) except RT (234) cpld_control_p8_gnu, which was terminated due to time limit. This is confirmed by the err log in (ii.). Lastly, cpld_control_p8_gnu was re-ran and completed successfully as confirmed in (iii.).
  1. FAIL-cpld_control_p8_gnu-RegressionTests_hera.log.txt
  2. TIMEOUT-cpld_control_p8_gnu-err.txt
  3. PASS-cpld_control_p8_gnu-RegressionTests_hera.log.txt

Testing Log (for CM's):

• RDHPCS
  • Intel
    • Hera
    • Orion
    • Jet
    • Gaea
    • Cheyenne
  • GNU
    • Hera
    • Cheyenne
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
  • Completed
• opnReqTest
  • N/A
  • Log attached to comment

[BrianCurtis-NOAA]

Hey @MatthewMasarik-NOAA, can you try the cpld_control_p8_gnu test with the changes in PR #1810 and see if it fixes your issue?

[MatthewMasarik-NOAA]

Hey @MatthewMasarik-NOAA, can you try the cpld_control_p8_gnu test with the changes in PR #1810 and see if it fixes your issue?

Hi @BrianCurtis-NOAA, sure thing. I'll let you know shortly.

[DeniseWorthen]

I'm curious why the re-test you did shows a wall clock time of 2234s. That test has been running right at the 30min limit (~1800s). Why is this test running so much slower w/ your changes?

Current develop branch:

  0: The total amount of wall time                        = 1757.157913
  0: The maximum resident set size (KB)                   = 1429100

Test 234 cpld_control_p8_gnu PASS

[MatthewMasarik-NOAA]

Why is this test running so much slower w/ your changes?

I'm not sure.. this is the first I'm aware of the timing. I'm going to try @BrianCurtis-NOAA's updates now and see what that gives.

[DeniseWorthen]

Because there has been an issue w/ hera.gnu, we've extended the wall clock for just hera.gnu for the coupled tests. It's curently set at 40mins, which has been working because the test usually requires just a smidge over the 30min wall clock default. I looked at the PR changes and I don't see anything obvious that would cause WW3 to run slower.

[MatthewMasarik-NOAA]

From tailing the output files during the simulation, it seemed like the initialization took awhile. Once the model started integrating time steps it marched along quickly.

[MatthewMasarik-NOAA]

The RT cpld_control_p8_gnu is re-running now with #1810 merged in, it will complete in ~30mins.

[MatthewMasarik-NOAA]

@BrianCurtis-NOAA @DeniseWorthen, with Brian's updates from #1810 the re-run completed in ~31mins (pr_1810-cpld_control_p8_gnu-out.txt). Ps, run directory /scratch1/NCEPDEV/stmp2/Matthew.Masarik/FV3_RT/rt_29811/cpld_control_p8_gnu.

@BrianCurtis-NOAA let me know if you want me to push the updates from your pr to this branch, or how to proceed

pr_1810-cpld_control_p8_gnu-RegressionTests_hera.log.txt

[BrianCurtis-NOAA]

We may get #1810 in sooner, but @jkbk2004 note that #1810 should go in before/with this PR.

[MatthewMasarik-NOAA]

Okay, sounds good.

[MatthewMasarik-NOAA]

Tests re-run on hera today with #1810 included for runtime fix (branch on my fork including pr 1810: MatthewMasarik-NOAA/ufs-weather-model/syncww3_2023jun27_incl_pr1810).

RegressionTests_hera.log.txt

[MatthewMasarik-NOAA]

Tests re-run after #1775 merged, 2023-jul-05. Contains ORT bug fix PR #1810.

2023-jul-05_RegressionTests_hera.log.txt

[MatthewMasarik-NOAA]

RegressionTests_hera.log.txt - all tests PASS.

[MatthewMasarik-NOAA]

RT's up-to-date with #1777 are currently running on orion.

[BrianCurtis-NOAA]

If it helps, you only need to run Hera logs once you're done making commits to your PR (not including bringing repositories up to date). If CM's do things correctly it should be the same once your PR hits the top of the commit queue.

[MatthewMasarik-NOAA]

If it helps, you only need to run Hera logs once you're done making commits to your PR (not including bringing repositories up to date). If CM's do things correctly it should be the same once your PR hits the top of the commit queue.

Okay, I can say that I haven't made any new commits, but I did just update the UFS repo after the merge. I submitted RT's to confirm all is well will my PR after the latest update. Am I understanding correctly that it's not necessary to re-run the RT's after merges?

[BrianCurtis-NOAA]

If it helps, you only need to run Hera logs once you're done making commits to your PR (not including bringing repositories up to date). If CM's do things correctly it should be the same once your PR hits the top of the commit queue.

Okay, I can say that I haven't made any new commits, but I did just update the UFS repo after the merge. I submitted RT's to confirm all is well will my PR after the latest update. Am I understanding correctly that it's not necessary to re-run the RT's after merges?

Correct, it is not required to re-run RT's after merges.

[MatthewMasarik-NOAA]

Correct, it is not required to re-run RT's after merges.

Great, I copy. Thank you for clarifying.

[zach1221]

Please see jenkins-ci logs for 1811. ORTs passed. We may begin Regression Testing now.
PR 1811 Jenkins-CI ORT log.pdf

[zach1221]

Ok, as discussed in our Code Managers meeting, it does look like I'm able to run tests on Cheyenne now. So, I'll post the log for those tests here as soon as possible.

[MatthewMasarik-NOAA]

Curious if there's an ETA for this merging today?

[zach1221]

Hi, @MatthewMasarik-NOAA , I'm attempting to get the RTs to complete on Cheyenne but have disk quota issues. @BrianCurtis-NOAA is it alright if we skip Cheyenne for this PR, and I'll work to get the issue resolved for the next?

[MatthewMasarik-NOAA]

Hi, @MatthewMasarik-NOAA , I'm attempting to get the RTs to complete on Cheyenne but have disk quota issues. @BrianCurtis-NOAA is it alright if we skip Cheyenne for this PR, and I'll work to get the issue resolved for the next?

Thanks @zach1221, no worries. I was just curious so I'd be nearby if needed.

[zach1221]

@MatthewMasarik-NOAA , understood. Well the WW3 sub-pr has been merged. Can you please update the submodule pointer and revert the change in gitmodules?
WW3 hash 59c554a12df3a04e0402ce5f17bb32cbbac193b2

[MatthewMasarik-NOAA]

@MatthewMasarik-NOAA , understood. Well the WW3 sub-pr has been merged. Can you please update the submodule pointer and revert the change in gitmodules?
WW3 hash 59c554a12df3a04e0402ce5f17bb32cbbac193b2

@zach1221, I believe this should be done now.

[zach1221]

Thanks, @MatthewMasarik-NOAA . Sending out final reviews now.