[release/public-v2.2.0] Update MacOS configuration and compatibility (#…

…953)

Updated modulefiles, scripts and configurations for running SRW v2.2.0 on MacOS platforms, summarized below. The changes include adaptation of the code for two MacOS architecthures, Intel/x86_64 and M1/arm64 (could be aarch64 if using Rosetta)
* modulefiles in ./modulefiles, ./modulefiles/tasks/macos/, srw_common.lua (successfully tested on all platforms but cloud)
* machine file in ./ush/machine/macos.yaml
* ./etc/lmod-setup.*
* scripts in ./ush/wrappers/* ./jobs/ , ./scripts/*.sh - the header changed to #!/usr/bin/env bash; a small bug in one of the scripts
* a patch file is applied for the ./sorc/CMakeLists.txt to build SRW on M1/arm64 machine; patch is located in ./patches/patch_macos_arm64_sorc_cmakelists.txt, and it is applied in ./devbuild.sh when needed
* small fixes in ./ush/job_preamble.sh, ./ush/preamble.sh (use mac-specific commands for runtime estimates)
* fix for ./ush/python_utils/filesys_cmds_vrfy.py: function vrfy_ln creates links with gnu-ln on MacOS (from coreutils)
* updates to the documentation for the MacOS users and adding options to install workflow conda environments.

---------

Co-authored-by: Natalie Perlin <Natalie.Perlin@noaa.gov>
Co-authored-by: Natalie Perlin <Natalie@Natalies-MacBook-Air.local>
Co-authored-by: Gillian Petro <96886803+gspetro-NOAA@users.noreply.github.com>

README.md

@@ -15,7 +15,7 @@ https://epic.noaa.gov/wp-content/uploads/2022/12/Debugging-Guide.pdf
 
 The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
 
-UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
+UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
 
 [![Python unittests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_unittests.yaml)
 [![Python functional tests](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml/badge.svg)](https://github.com/ufs-community/ufs-srweather-app/actions/workflows/python_func_tests.yaml)

devbuild.sh

@@ -417,6 +417,19 @@ else
 fi
 module list
 
+# Apply patch for sorc/CMakeLists.txt for MacOS arm64/aarch64
+if [[ "${PLATFORM}" == "macos" ]]; then
+  ARCH=$(uname -m)
+  if [[ "${ARCH}" == arm64 ]] || [[ "${ARCH}" == aarch64 ]]; then
+    patch1="sorc/patch_macos_arm64_sorc_cmakelists.txt"
+    if patch -p1 -R --dry-run --silent -d ./sorc -N < ${patch1} 1> /dev/null; then
+      echo "Patch ${patch1} was already applied";
+    else
+      patch -p1 -d ./sorc -N < ${patch1}
+    fi
+  fi
+fi
+
 mkdir -p ${BUILD_DIR}
 cd ${BUILD_DIR}
 

docs/UsersGuide/source/BackgroundInfo/Introduction.rst

@@ -31,7 +31,7 @@ Since the last release, developers have added a variety of features:
 
 The SRW App v2.2.0 citation is as follows and should be used when presenting results based on research conducted with the App:
 
-UFS Development Team. (2023, Oct. 30). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
+UFS Development Team. (2023, Oct. 31). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10015544
 
 User's Guide Organization 
 ============================

docs/UsersGuide/source/BuildingRunningTesting/BuildSRW.rst

@@ -15,7 +15,7 @@ The Unified Forecast System (:term:`UFS`) Short-Range Weather (SRW) Application
 
 To build the SRW App, users will complete the following steps:
 
-   #. :ref:`Install prerequisites <HPCstackInfo>`
+   #. :ref:`Install prerequisites <StackInfo>`
    #. :ref:`Clone the SRW App from GitHub <DownloadSRWApp>`
    #. :ref:`Check out the external repositories <CheckoutExternals>`
    #. :ref:`Set up the build environment and build the executables <BuildExecutables>`
@@ -358,9 +358,8 @@ Additional Details for Building on MacOS or Generic Linux
 
 The SRW App can be built on MacOS and generic Linux machines after the prerequisite software has been installed on these systems (via :term:`HPC-Stack` or :term:`spack-stack`). The installation for MacOS is architecture-independent and has been tested using both x86_64 and M1 chips (running natively). The following configurations for MacOS have been tested:
 
-   #. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), Monterey Sur 12.1, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
-   #. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, Big Sur 11.6.4, GNU compiler suite v.11.3.0 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
-   #. MacBook Pro 2015, 2.8 GHz Quad-Core Intel Core i7 (x86_64), Catalina OS X 10.15.7, GNU compiler suite v.11.2.0_3 (gcc, gfortran, g++); mpich 3.3.2 or openmpi/4.1.2
+   #. MacBookPro 2019, 2.4 GHz 8-core Intel Core i9 (x86_64), OS Monterey 12.6.1, 32 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
+   #. MacBookAir 2020, M1 chip (arm64, running natively), 4+4 cores, OS Ventura 13.0.1, 16 GB RAM; GNU compiler suite v.12.3.0 (gcc, gfortran, g++); openmpi/4.1.5
 
 Several Linux builds have been tested on systems with x86_64 architectures.
 

docs/UsersGuide/source/BuildingRunningTesting/ContainerQuickstart.rst

@@ -80,19 +80,23 @@ On most Level 1 systems, a container named ``ubuntu20.04-intel-srwapp-release-pu
 
    * - Machine
      - File Location
-   * - Cheyenne/Derecho
+   * - Cheyenne/Derecho `*`_
      - /glade/scratch/epicufsrt/containers
-   * - Gaea
+   * - Gaea `*`_
      - /lustre/f2/dev/role.epic/containers
    * - Hera
      - /scratch1/NCEPDEV/nems/role.epic/containers
    * - Jet
      - /mnt/lfs4/HFIP/hfv3gfs/role.epic/containers
    * - NOAA Cloud
      - /contrib/EPIC/containers
-   * - Orion/Hercules
+   * - Orion/Hercules `*`_
      - /work/noaa/epic/role-epic/contrib/containers
 
+.. _`*` : 
+
+   \* On these systems, container testing shows inconsistent results. 
+
 .. note::
    * On Gaea, Singularity/Apptainer is only available on the C5 partition, and therefore container use is only supported on Gaea C5. 
    * The NOAA Cloud containers are accessible only to those with EPIC resources. 

docs/UsersGuide/source/BuildingRunningTesting/RunSRW.rst

@@ -192,29 +192,100 @@ MacOS requires the installation of a few additional packages and, possibly, an u
 Creating the |wflow_env| Environment on Linux and Mac OS
 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
 
-On generic Mac and Linux systems, users need to create a conda |wflow_env| environment. The environment can be stored in a local path, which could be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) The following is a brief recipe for creating a virtual conda environment on non-Level 1 platforms. It uses the aarch64 (64-bit ARM) Miniforge for Linux and installs into $HOME/conda. Adjust as necessary for your target system.
+On generic Mac and Linux systems, users need to create a conda |wflow_env| environment that contains python packages required for running the workflow. Other conda environments may need to be activated for running graphics generation tasks (|graphics_env|) or when testing the AQM/CMAQ (|cmaq_env|). Python packages in these other environments may conflict with those in |wflow_env|. The environments can be stored in a local path, which can be a default location or a user-specified location (e.g., ``$HOME/condaenv/venvs/`` directory). (To determine the default location, use the ``conda info`` command, and look for the ``envs directories`` list.) 
+These conda environments can be added to the existing python or conda modules.
 
-.. code-block:: console
+There are several options available for building virtual conda environments on non-Level 1 platforms. The examples in this section use the aarch64 (64-bit ARM) Miniforge for Linux and install into ``$HOME/conda``. Users should adjust as needed for their target system. 
 
-   wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
-   bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
-   rm Miniforge3-Linux-aarch64.sh
-   source ~/conda/etc/profile.d/conda.sh
-   conda activate
-   conda install -y conda-build conda-verify
-   cd path/to/your/workflow-tools/clone
-   conda build recipe
-   conda create -y -n workflow_tools -c local workflow_tools
-   conda activate workflow_tools
+**Options:**
 
-In future shells, you can activate and use this environment with:
 
-.. code-block:: console
+1) Users can add the following environment .yaml files: 
+
+   a) workflow_tools.yaml for |wflow_env|
+
+   .. code-block:: console
+
+      name: workflow_tools
+      channels:
+        - conda-forge
+        - defaults
+      dependencies:
+        - python=3.9*
+        - boto3=1.22*
+        - black
+        - f90nml=1.4*
+        - jinja2=3.0*
+        - numpy=1.21*
+        - pylint
+        - pytest
+        - pyyaml=6.0*
+        - tox  
+
+   b) regional_workflow.yaml for |graphics_env|
+
+   .. code-block:: console
+   
+      name: regional_workflow
+      channels:
+        - conda-forge
+        - defaults
+      dependencies:
+        - python=3.9.*
+        - f90nml
+        - jinja2
+        - pyyaml
+        - scipy
+        - matplotlib=3.5.2*
+        - pygrib
+        - cartopy
+   
+   c) regional_workflow_cmaq.yaml for |cmaq_env|
+
+   .. code-block:: console
+   
+      name: regional_workflow_cmaq
+      channels:
+        - conda-forge
+        - defaults
+      dependencies:
+        - python=3.9.12
+        - f90nml=1.4*
+        - jinja2=3.0*
+        - pyyaml=6.0*
+        - scipy
+        - matplotlib
+        - pygrib
+        - cartopy
+        - netcdf4
+        - xarray
+
+2) Users can instead install Miniforge. This example uses the aarch64 (64-bit ARM) Miniforge for Linux that installs into ``$HOME/conda``. Users should adjust as needed for their target system. 
+
+   .. code-block:: console
+   
+      wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
+      bash Miniforge3-Linux-aarch64.sh -bfp ~/conda
+      rm Miniforge3-Linux-aarch64.sh
+      source ~/conda/etc/profile.d/conda.sh
+      conda activate
+      conda install -y conda-build conda-verify
+      cd path/to/your/workflow-tools/clone
+      conda build recipe
+      conda create -y -n workflow_tools -c local workflow_tools
+      conda activate workflow_tools
+
+   In future shells, you can activate and use this environment with:
+
+   .. code-block:: console
+   
+      source ~/conda/etc/profile.d/conda.sh
+      conda activate workflow_tools
+   
+   See the `workflow-tools repository <https://github.com/ufs-community/workflow-tools>`__ for additional documentation. 
 
-   source ~/conda/etc/profile.d/conda.sh
-   conda activate workflow_tools
+3) A third option is to build miniconda3 and create an Lmod modulefile that can be loaded with other modules during the workflow. The module can be added to the user's ``wflow_<platform>.lua`` modulefile, and the environments can be activated or deactivated as needed for a particular workflow task. A repository with full installation instructions, a modulefile template, and environment configuration files can be accessed in `NOAA-EPIC/miniconda3 repository <https://github.com/NOAA-EPIC/miniconda3>`__. Full instructions can be viewed in the `README.md file <https://github.com/NOAA-EPIC/miniconda3/edit/master/README.md>`__. 
 
-See the `workflow-tools repository <https://github.com/ufs-community/workflow-tools>`__ for additional documentation. 
 
 Modify a ``wflow_<platform>`` File
 ``````````````````````````````````````
@@ -632,7 +703,7 @@ This can be helpful when conducting multiple experiments with different types of
 Plotting Configuration (optional)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
+An optional Python plotting task (plot_allvars) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
 output over the :term:`CONUS`. It generates graphics plots for a number of variables, including:
 
    * 2-m temperature

docs/UsersGuide/source/CustomizingTheWorkflow/InputOutputFiles.rst

@@ -164,8 +164,8 @@ If users wish to modify the fields or levels that are output from the UPP, they
    This process requires advanced knowledge of which fields can be output for the UFS Weather Model.
 
 UPP Product Output Tables for the UFS SRW LAM Grid:
-   * :doc:`3D Native Hybrid Level Fields <tables/SRW_NATLEV_table>`
-   * :doc:`3D Pressure Level Fields <tables/SRW_PRSLEV_table>`
+   * :ref:`3D Native Hybrid Level Fields <SRW_NATLEV_table>`
+   * :ref:`3D Pressure Level Fields <SRW_PRSLEV_table>`
 
 Use the instructions in the `UPP User's Guide <https://upp.readthedocs.io/en/upp-srw-v2.2.0-docs/InputsOutputs.html#control-file>`__ to make modifications to the ``fv3lam.xml`` file and to remake the flat text file, called ``postxconfig-NT-fv3lam.txt`` (default), that the UPP reads.
 

docs/UsersGuide/source/conf.py

@@ -89,6 +89,8 @@
 
 rst_prolog = """
 .. |wflow_env| replace:: ``workflow_tools``
+.. |graphics_env| replace:: ``regional_workflow``
+.. |cmaq_env| replace:: ``regional_workflow_cmaq``
 .. |activate| replace:: ``conda activate workflow_tools``
 .. |prompt| replace:: ``(workflow_tools)``
 .. |latestr| replace:: v2.2.0

docs/UsersGuide/source/tables/SRW_NATLEV_table.rst

@@ -1,5 +1,7 @@
 :orphan:
 
+.. _SRW_NATLEV_table:
+
 ************************************************************
 Fields Requested in the UPP Parameter Table for SRW NATLEV
 ************************************************************

docs/UsersGuide/source/tables/SRW_PRSLEV_table.rst

@@ -1,5 +1,7 @@
 :orphan:
 
+.. _SRW_PRSLEV_table:
+
 **********************************************************
 Fields Requested in the UPP Parameter Table for SRW PRSLEV
 **********************************************************

etc/lmod-setup.csh

@@ -18,22 +18,23 @@ if ( "$L_MACHINE" != wcoss2 ) then
 endif
 
 if ( "$L_MACHINE" == macos ) then
-   arch=$(uname -m)
-   [[ "$arch" = arm64 ]] && export ENV="/opt/homebrew/opt/lmod/init/csh"
-   [[ "$arch" = x86_64 ]] && export ENV="/usr/local/opt/lmod/init/csh"
-   source $ENV
+   if ( -d /opt/homebrew/opt/lmod ) then
+       source /opt/homebrew/opt/lmod/init/csh
+   else if ( -d /usr/local/opt/lmod ) then
+       source /usr/local/opt/lmod/init/csh
+   else
+       echo "Path for Lmod is unknown, verify whether Lmod is installed"
+   endif
 
    module purge
 
 else if ( "$L_MACHINE" == linux ) then
-   setenv ENV "/usr/share/lmod/lmod/init/csh"
-   source $ENV
+   source /usr/share/lmod/lmod/init/csh
 
    module purge
 
 else if ( "$L_MACHINE" == singularity ) then
-   set ENV="/usr/share/lmod/lmod/init/csh"
-   source $ENV
+   source /usr/share/lmod/lmod/init/csh
 
    module purge
 

etc/lmod-setup.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/usr/bin/env bash
 
 if [ $# = 0 ]; then
    L_MACHINE=${MACHINE}
@@ -26,10 +26,12 @@ fi
 
 if [ "$L_MACHINE" = macos ]; then
    arch=$(uname -m)
-   [[ "$arch" = arm64 ]] && export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
-   [[ "$arch" = x86_64 ]] && export BASH_ENV="/usr/local/opt/lmod/init/bash"
+   if [ "$arch" = arm64 ] || [ "$arch" = aarch64 ]; then
+      export BASH_ENV="/opt/homebrew/opt/lmod/init/bash"
+   else
+      export BASH_ENV="/usr/local/opt/lmod/init/bash"
+   fi
    source $BASH_ENV
-
    module purge
 
 elif [ "$L_MACHINE" = linux ]; then

jobs/JREGIONAL_AQM_ICS

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_AQM_LBCS

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_BIAS_CORRECTION_O3

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_BIAS_CORRECTION_PM25

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_CHECK_POST_OUTPUT

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_FIRE_EMISSION

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_GRID

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_ICS

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_LBCS

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_OROG

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_MAKE_SFC_CLIMO

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_EMISSION

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_GFS_SFC

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_NEXUS_POST_SPLIT

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------

jobs/JREGIONAL_PLOT_ALLVARS

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 #
 #-----------------------------------------------------------------------