[develop] Modify get_lbcs and make_lbcs.

[danielabdi-noaa]

DESCRIPTION OF CHANGES:

Addresses issue #665. Get_lbcs need to download 0th hour LBCS and make_lbcs need to be parallelized.
Note that there is no workflow so this PR can not be run independently until then.
However, it did successfully run for both spinup/production cycles in PR #540

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
  Successfully run these tasks modified for RRFS in PR [develop] Incorporate RRFS_dev1 workflow to SRW and import most tasks #540
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

• [develop] Add new RRFS variables such as NWGES and workflow control variables #647

DOCUMENTATION:

ISSUE:

• Download 0th hour LBCS and parallelize make_lbcs for RRFS #665

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

[panll]

Tested on Hera. Running time error:
.../ufs-srweather-app/jobs/JREGIONAL_MAKE_ICS: line 63: NWGES_DIR: unbound variable
End JREGIONAL_MAKE_ICS at Tue Apr 11 17:42:24 UTC 2023 with error code 1 (time elapsed: 00:00:00)

[danielabdi-noaa]

@panll I have added the necessary NWGES variables but this PR requires a workflow update for the specific way RRFS downloads LBCS, so it can not be tested without that.

[christinaholtNOAA]

Daniel, thanks for preparing this PR. I've left some comments below that focus on making sure that we don't change existing behavior while capturing RRFS configuration settings, and that the mods stay in scope for the intent of this PR.

[venitahagerty]

Machine: hera
Compiler: intel
Job: WE
Repo location: /scratch1/BMC/zrtrr/rrfs_ci/autoci/pr/1271094432/20230425193511/ufs-srweather-app
Build was Successful
Rocoto jobs started
Experiment done: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
2023-04-25 20:13:24 +0000 :: hfe07 :: This cycle is complete: Success
Experiment failed: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
2023-04-25 20:27:14 +0000 :: hfe07 :: Task run_MET_PcpCombine_fcst_APCP01h_mem000, jobid=44194882, in state DEAD (FAILED), ran for 13.0 seconds, exit status=256, try=2 (of 2)
Experiment done: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
2023-04-25 20:11:40 +0000 :: hfe07 :: This cycle is complete: Success
Experiment done: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
2023-04-25 20:07:58 +0000 :: hfe07 :: This cycle is complete: Success
Experiment done: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional_plot
2023-04-25 20:09:30 +0000 :: hfe07 :: This cycle is complete: Success
Experiment done: pregen_grid_orog_sfc_climo
2023-04-25 20:07:17 +0000 :: hfe07 :: This cycle is complete: Success
Experiment done: grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
2023-04-25 20:07:31 +0000 :: hfe07 :: This cycle is complete: Success
If test failed, please make changes and add the following label back:
ci-hera-intel-WE

[MichaelLueken]

@danielabdi-noaa Overall, these changes look good to me. I did want to point out a potential issue between NCO implementation and symlinks. NCO created a Bugzilla ticket to remove symlinks from the global workflow. From Steven Earle:

The issue described in the ticket goes much further than just the recent wave issue; I can update the bugzilla if that's desired.
DATA needs to be self contained for several reasons....
-- Portability
-- Contained IO, making management of the system/storage possible
-- Pristine place to save after failures for debug/troubleshooting later

While the use of symlinks are fine in develop, it is possible that NCO will require that they are replaced with copy/move/rsync in operations. While this work can be done in the release branch for RRFS, it is still something to keep in mind moving forward.

[danielabdi-noaa]

@MichaelLueken Thanks, I agree with your assessment of use of symlinks. The NCO implementation here that uses NWGES does not conform with latest NCO standards, however, RRFS_dev1 uses it throughout. It was decided a couple of months ago to use it as a temporary solution, but the plan was indeed to get rid of the NWGES directory and thus associated symlinks in the future.

[panll]

Tested on Hera, it works fine.

[danielabdi-noaa]

@christinaholtNOAA Is there something I need to address other than the potential AQM issue? I did run the make_lbcs test with an AQM test case a while ago and it seems to work.

[christinaholtNOAA]

I'm sorry for the delay in re-reviewing. I've left a few more comments below. It seems we're still waiting on @chan-hoo to weigh in, as well.

[MichaelLueken]

@danielabdi-noaa There were a few failures in the Jenkins tests, but they were expected failures:

On Cheyenne GNU - the nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km test failed intermittently in the run_post tasks. This has been an issue since the test has been introduced.

Following the PrgEnv update on Gaea, it is currently impossible to build and run the SRW and weather model on the machine. Working with Natalie to get this corrected as soon as possible.

On Hera Intel - the grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta test failed in run_fcst (issue #731).

Since the failures are expected, I will now move forward with merging this PR to develop.