ufs-community · zach1221 · Oct 24, 2023 · Sep 20, 2023 · Sep 21, 2023 · Sep 21, 2023
diff --git a/.gitmodules b/.gitmodules
@@ -1,7 +1,7 @@
 [submodule "FV3"]
   path = FV3
-  url = https://github.com/NOAA-EMC/fv3atm
-  branch = develop
+  url = https://github.com/mdtoyNOAA/fv3atm
+  branch = develop_meso_fix_new_ksmax
 [submodule "WW3"]
   path = WW3
   url = https://github.com/NOAA-EMC/WW3

diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -4,6 +4,10 @@
 
 # Required
 version: 2
+build: 
+  os: ubuntu-20.04
+  tools: 
+    python: "3.9"
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
@@ -18,6 +22,5 @@ formats: all
 
 # Optionally set the version of Python and requirements required to build your docs
 python:
-  version: 3.7
   install:
     - requirements: doc/UsersGuide/requirements.txt
diff --git a/FV3 b/FV3
diff --git a/doc/UsersGuide/source/AutomatedTesting.rst b/doc/UsersGuide/source/AutomatedTesting.rst
@@ -20,7 +20,7 @@ The UFS Weather Model (:term:`WM`) uses GitHub Actions (GHA), a GitHub-hosted co
 to perform CI/CD testing. Build jobs are done on GHA-provided virtual machines. Test jobs are 
 performed on the Amazon Web Services (AWS) cloud platform using a number of EC2 instances. 
 Builds and tests are carried out in a Docker container. The container includes a pre-installed version of the
-:term:`HPC-Stack`, which includes all prerequisite libraries. Input data needed to run the tests 
+:term:`spack-stack`, which includes all prerequisite libraries. Input data needed to run the tests 
 are stored as a separate Docker container.
 
 When a developer makes a pull request (PR) to the UFS WM repository, a code
@@ -31,14 +31,14 @@ The CI/CD workflow then executes the following steps:
       subcomponents are up to date with the top of the ``develop`` branch.
 
    #. If the check is successful, build jobs are started on GHA-provided virtual machines
-      by downloading the hpc-stack Docker container stored in Docker Hub.
+      by downloading the spack-stack Docker container stored in Docker Hub.
 
    #. Once all build jobs are successful, the created executable files are stored as
       artifacts in GHA.
 
    #. A number of AWS EC2 instances are started.
 
-   #. Test jobs are started on AWS after downloading the hpc-stack Docker container,
+   #. Test jobs are started on AWS after downloading the spack-stack Docker container,
       the executable file from the build job, and the input-data Docker container.
 
    #. When all tests are complete, EC2 instances are stopped. Test results are reported

diff --git a/doc/UsersGuide/source/BuildingAndRunning.rst b/doc/UsersGuide/source/BuildingAndRunning.rst
@@ -14,15 +14,15 @@ through NOAA and its affiliates. These systems are named (e.g., Hera, Orion, Che
 Level 3 & 4 systems include certain personal computers or non-NOAA-affiliated HPC systems. 
 The prerequisite software libraries for building the WM already exist in a centralized location on Level 1/preconfigured 
 systems, so users may skip directly to :ref:`getting the data <GetData>` and downloading the code. 
-On other systems, users will need to build the prerequisite libraries using :term:`HPC-Stack` or :term:`spack-stack`. 
+On other systems, users will need to build the prerequisite libraries using :term:`spack-stack` or :term:`HPC-Stack`. 
 
 =======================
 Prerequisite Libraries
 =======================
 
 The UFS WM requires a number of libraries.
 The WM uses two categories of libraries, which are available as a bundle via 
-:term:`HPC-Stack` or :term:`spack-stack`:
+:term:`spack-stack` or :term:`HPC-Stack`:
 
    #. :term:`NCEP` libraries (:term:`NCEPLIBS`): These are libraries developed for use with NOAA weather models.
       Most have an NCEPLIBS prefix in the repository (e.g., NCEPLIBS-bacio). Select tools from the UFS
@@ -34,11 +34,12 @@ The WM uses two categories of libraries, which are available as a bundle via
       instead. 
 
 .. note::
-   Currently, HPC-Stack is the software stack validated by the UFS WM for running :term:`regression tests <RT>`. 
-   However, UFS applications are shifting to spack-stack, which is a Spack-based 
-   method for installing UFS prerequisite software libraries. The spack-stack is currently 
-   used on NOAA Cloud platforms and in containers, while HPC-Stack is still used on NOAA
-   Research & Development HPC Systems (RDHPCS). 
+   Currently, spack-stack is the software stack validated by the UFS WM for running 
+   :term:`regression tests <RT>`. Spack-stack is a Spack-based method for installing UFS 
+   prerequisite software libraries. UFS applications and components are also shifting to 
+   spack-stack from HPC-Stack but are at various stages of this transition. 
+   Although users can still build and use HPC-Stack, the UFS WM no longer uses HPC-Stack 
+   for validation, and support for this option is being deprecated. 
 
 ----------------
 Common Modules
@@ -70,8 +71,8 @@ The most updated list of common modules can be viewed in ``ufs_common.lua``
 `here <https://github.com/ufs-community/ufs-weather-model/blob/develop/modulefiles/ufs_common.lua>`__.
 
 .. attention::
-   Documentation is available for installing `HPC-Stack <https://hpc-stack.readthedocs.io/en/latest/>`__ 
-   and `spack-stack <https://spack-stack.readthedocs.io/en/latest/>`__, respectively. 
+   Documentation is available for installing `spack-stack <https://spack-stack.readthedocs.io/en/latest/>`__
+   and `HPC-Stack <https://hpc-stack.readthedocs.io/en/latest/>`__, respectively. 
    One of these software stacks (or the libraries they contain) must be installed before running the UFS Weather Model. 
 
 .. _GetData:

diff --git a/doc/UsersGuide/source/CodeOverview.rst b/doc/UsersGuide/source/CodeOverview.rst
@@ -19,16 +19,16 @@ Four levels of support have been defined for :term:`UFS` applications, and the U
 
 Level 1 Systems
 ==================
-Preconfigured (Level 1) systems for the UFS WM already have the required external libraries available in a central location via :term:`HPC-Stack` or :term:`spack-stack`. The WM is expected to build and run out-of-the-box on these systems, and users can download the WM code without first installing prerequisite software. Additionally, regression test data is already available on these systems. In general, users must have access to these Level 1 systems in order to use them.
+Preconfigured (Level 1) systems for the UFS WM already have the required external libraries available in a central location via :term:`spack-stack`. The WM is expected to build and run out-of-the-box on these systems, and users can download the WM code without first installing prerequisite software. Additionally, regression test data is already available on these systems. In general, users must have access to these Level 1 systems in order to use them.
 
 Currently, Level 1 (or Tier-1) platforms for regression testing are: 
 
    * WCOSS2 (Intel)
-   * Cheyenne (Intel/GNU compilers)
    * Gaea (Intel)
    * Hera (Intel/GNU compilers)
    * Jet (Intel)
    * Orion (Intel)
+   * Hercules (Intel/GNU compilers)
    * AWS Docker container (Intel)
 
 More information is available in the `UFS WM wiki <https://github.com/ufs-community/ufs-weather-model/wiki/Regression-Test-Policy-for-Weather-Model-Platforms-and-Compilers>`__. 

diff --git a/doc/UsersGuide/source/Glossary.rst b/doc/UsersGuide/source/Glossary.rst
@@ -120,11 +120,11 @@ Glossary
 
    NCEPLIBS
       The software libraries created and maintained by :term:`NCEP` that are required for running 
-      :term:`chgres_cube`, the UFS Weather Model, and the :term:`UPP`. They are included in the `HPC-Stack <https://github.com/NOAA-EMC/hpc-stack>`__ and in `spack-stack <https://github.com/NOAA-EMC/spack-stack>`__. 
+      :term:`chgres_cube`, the UFS Weather Model, and the :term:`UPP`. They are included in `spack-stack <https://github.com/NOAA-EMC/spack-stack>`__ and `HPC-Stack <https://github.com/NOAA-EMC/hpc-stack>`__. 
 
    NCEPLIBS-external
       A collection of third-party libraries required to build :term:`NCEPLIBS`, :term:`chgres_cube`, 
-      the UFS Weather Model, and the :term:`UPP`. They are included in the :term:`HPC-Stack` and in :term:`spack-stack`.  
+      the UFS Weather Model, and the :term:`UPP`. They are included in :term:`spack-stack` and :term:`HPC-Stack`.  
 
    NEMS
       The NOAA Environmental Modeling System is a common modeling framework whose purpose is 

diff --git a/doc/UsersGuide/source/conf.py b/doc/UsersGuide/source/conf.py
@@ -41,6 +41,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'sphinx_rtd_theme',
     'sphinx.ext.autodoc',
     'sphinx.ext.doctest',
     'sphinx.ext.intersphinx',

diff --git a/tests/default_vars.sh b/tests/default_vars.sh
@@ -454,6 +454,9 @@ export DO_GSL_DRAG_SS=.false.
 export DO_GSL_DRAG_TOFD=.false.
 export DO_UGWP_V1=.false.
 export DO_UGWP_V1_OROG_ONLY=.false.
+export KNOB_UGWP_DOKDIS=1
+export KNOB_UGWP_NDX4LH=1
+
 
 # resolution dependent settings
 export CDMBWD_c48='0.071,2.1,1.0,1.0'

diff --git a/tests/parm/cpld_control.nml.IN b/tests/parm/cpld_control.nml.IN
@@ -37,8 +37,9 @@
   n_sponge = 42
   nudge_qv = .true.
   nudge_dz = .false.
-  tau = 10.0
-  rf_cutoff = 7.5e2
+  tau = 0.0
+  fast_tau_w_sec = 0.2
+  rf_cutoff = 10.
   d2_bg_k1 = @[D2_BG_K1]
   d2_bg_k2 = @[D2_BG_K2]
   kord_tm = -9
@@ -52,7 +53,7 @@
   a_imp = 1.
   p_fac = 0.1
   k_split = 2
-  n_split = 6
+  n_split = 5
   nwat = 6
   na_init = @[NA_INIT]
   d_ext = 0.
@@ -92,7 +93,6 @@
   agrid_vel_rst = .true.
   read_increment = @[READ_INCREMENT]
   res_latlon_dynamics = @[RES_LATLON_DYNAMICS]
-
 /
 
 &external_ic_nml
@@ -248,18 +248,27 @@
 /
 
 &cires_ugwp_nml
-  knob_ugwp_solver  = 2
-  knob_ugwp_source  = 1,1,0,0
-  knob_ugwp_wvspec  = 1,25,25,25
-  knob_ugwp_azdir   = 2,4,4,4
-  knob_ugwp_stoch   = 0,0,0,0
-  knob_ugwp_effac   = 1,1,1,1
-  knob_ugwp_doaxyz  = 1
-  knob_ugwp_doheat  = 1
-  knob_ugwp_dokdis  = 1
-  knob_ugwp_ndx4lh  = 1
-  knob_ugwp_version = @[KNOB_UGWP_VERSION]
+  knob_ugwp_solver   = 2
+  knob_ugwp_source   = 1,1,0,0
+  knob_ugwp_wvspec   = 1,25,25,25
+  knob_ugwp_azdir    = 2,4,4,4
+  knob_ugwp_stoch    = 0,0,0,0
+  knob_ugwp_effac    = 1,1,1,1
+  knob_ugwp_doaxyz   = 1
+  knob_ugwp_doheat   = 1
+  knob_ugwp_dokdis   = @[KNOB_UGWP_DOKDIS]
+  knob_ugwp_ndx4lh   = @[KNOB_UGWP_NDX4LH]
+  knob_ugwp_version  = @[KNOB_UGWP_VERSION]
   launch_level      = 54
+  knob_ugwp_palaunch = 275.0e2
+  knob_ugwp_nslope   = @[KNOB_UGWP_NSLOPE]
+  knob_ugwp_lzmax    = 15.750e3
+  knob_ugwp_lzmin    = 0.75e3
+  knob_ugwp_lzstar   = 2.0e3
+  knob_ugwp_taumin   = 0.25e-3
+  knob_ugwp_tauamp   = 3.0e-3
+  knob_ugwp_lhmet    = 200.0e3
+  knob_ugwp_orosolv  = 'pss-1986'
 /
 
 &gfdl_cloud_microphysics_nml

diff --git a/tests/rt.conf b/tests/rt.conf
@@ -21,10 +21,18 @@
 
 ### Intel Tests ###
 ### S2S tests ###
-COMPILE | s2swa_32bit  | intel | -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 |          | fv3 |
+COMPILE | s2swa_32bit  | intel | -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8,FV3_GFS_v17_coupled_p8_ugwpv1 |          | fv3 |
 RUN | cpld_control_p8_mixedmode                         |    - noaacloud                     | baseline |
-RUN | cpld_control_gfsv17                               |                                    | baseline |
-RUN | cpld_control_gfsv17_iau                           |                                    | baseline | cpld_control_gfsv17
+
+#GFS
+COMPILE | s2swa_32bit_pdlib  | intel | -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_ugwpv1 -DPDLIB=ON | + hera orion cheyenne wcoss2 acorn | fv3 |
+RUN | cpld_control_gfsv17                               | + hera orion cheyenne wcoss2 acorn | baseline |
+RUN | cpld_control_gfsv17_iau                           | + hera orion cheyenne wcoss2 acorn | baseline | cpld_control_gfsv17
+RUN | cpld_restart_gfsv17                               | + hera orion cheyenne wcoss2 acorn |          | cpld_control_gfsv17 
+RUN | cpld_mpi_gfsv17                                   | + hera orion cheyenne wcoss2 acorn |          | cpld_control_gfsv17
+
+COMPILE | s2swa_32bit_pdlib_debug | intel | -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_ugwpv1 -DPDLIB=ON -DDEBUG=ON | + hera orion cheyenne wcoss2 acorn | fv3 |
+RUN | cpld_debug_gfsv17                                 | + hera orion cheyenne wcoss2 acorn | baseline |
 
 COMPILE | s2swa | intel | -DAPP=S2SWA -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 | | fv3 |
 RUN | cpld_control_p8                                   |   - noaacloud                       | baseline |
@@ -73,7 +81,7 @@ COMPILE | s2sw_pdlib_debug | intel | -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_couple
 RUN | cpld_debug_pdlib_p8                               | + hera orion cheyenne wcoss2    | baseline |
 
 ### ATM tests ###
-COMPILE | atm_dyn32 | intel | -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON | | fv3 |
+COMPILE | atm_dyn32 | intel | -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn,FV3_GFS_v17_p8_ugwpv1 -D32BIT=ON | | fv3 |
 RUN | control_flake                                     |                            | baseline |
 RUN | control_CubedSphereGrid                           |                            | baseline |
 RUN | control_CubedSphereGrid_parallel                  | - noaacloud                | baseline |
@@ -89,6 +97,7 @@ RUN | control_lndp                                      |
 RUN | control_iovr4                                     |                            | baseline |
 RUN | control_iovr5                                     |                            | baseline |
 RUN | control_p8                                        | - noaacloud                | baseline |
+RUN | control_p8_ugwpv1                                 | - noaacloud                | baseline |
 RUN | control_restart_p8                                | - noaacloud                |          | control_p8
 RUN | control_qr_p8                                     | - noaacloud                |          |
 RUN | control_restart_qr_p8                             | - noaacloud                |          | control_qr_p8
@@ -308,11 +317,12 @@ RUN | regional_atmaq_faster                             | - jet wcoss2 acorn s4
 
 ### GNU TESTS ###
 ### CCPP PROD tests ###
-COMPILE | atm | gnu | -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v16_ras,FV3_GFS_v17_p8 | + hera cheyenne hercules | fv3 |
+COMPILE | atm | gnu | -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v16_ras,FV3_GFS_v17_p8,FV3_GFS_v17_p8_ugwpv1 | + hera cheyenne hercules | fv3 |
 RUN | control_c48                                       | + hera cheyenne hercules            | baseline |
 RUN | control_stochy                                    | + hera cheyenne hercules            | baseline |
 RUN | control_ras                                       | + hera cheyenne hercules            | baseline |
 RUN | control_p8                                        | + hera cheyenne hercules            | baseline |
+RUN | control_p8_ugwpv1                                 | + hera cheyenne hercules            | baseline |
 RUN | control_flake                                     | + hera cheyenne hercules            | baseline |
 
 COMPILE | rrfs | gnu | -DAPP=ATM -DCCPP_SUITES=FV3_RAP,FV3_RAP_sfcdiff,FV3_HRRR,FV3_RRFS_v1beta -D32BIT=ON | + hera cheyenne hercules | fv3 |