Merge pull request #2513 from samsrabin/merge-b4bdev-20240502

Merge b4b-dev branch into master.

.git-blame-ignore-revs

@@ -2,6 +2,7 @@
 b429b63824e09f82e95d2982f14311cbbd8e4a37
 d229b5c6689efc4c2a6cef077515c4ccd5c18ff6
 4cd83cb3ee6d85eb909403487abf5eeaf4d98911
+d229b5c6689efc4c2a6cef077515c4ccd5c18ff6
 0aa2957c1f8603c63fa30b11295c06cfddff44a5
 2cdb380febb274478e84cd90945aee93f29fa2e6
 e44dc469439e02e9ee582dab274d890ebdfab104

.github/PULL_REQUEST_TEMPLATE.md

@@ -10,6 +10,8 @@ Are answers expected to change (and if so in what way)?
 
 Any User Interface Changes (namelist or namelist defaults changes)?
 
+Does this create a need to change or add documentation? Did you do so?
+
 Testing performed, if any:
 (List what testing you did to show your changes worked as expected)
 (This can be manual testing or running of the different test suites)

bld/namelist_files/namelist_defaults_ctsm.xml

@@ -1395,6 +1395,8 @@ lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_ne16np4.pg3_hist_2000_78pfts_c240216.n
 <fsurdat hgrid="0.125nldas2" sim_year="2000">
 lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_0.125nldas2_hist_2000_78pfts_c240216.nc</fsurdat>
 
+<fsurdat hgrid="64x128"      sim_year="2000">
+lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_T42_hist_2000_78pfts_c240425.nc</fsurdat>
 <fsurdat hgrid="360x720cru"  sim_year="2000">
 lnd/clm2/surfdata_esmf/ctsm5.2.0/surfdata_360x720cru_hist_2000_78pfts_c240216.nc</fsurdat>
 <fsurdat hgrid="C96"          sim_year="2000">

cime_config/usermods_dirs/f09_37x288pt_PanBoreal/shell_commands

@@ -0,0 +1,8 @@
+
+# Change below line if you move the subset data directory
+
+./xmlchange CLM_USRDAT_DIR='$DIN_LOC_ROOT/lnd/clm2/regional_datasets/f09_38x288pt_PanBoreal'
+
+./xmlchange ATM_DOMAIN_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'
+./xmlchange LND_DOMAIN_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'
+./xmlchange MASK_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'

cime_config/usermods_dirs/f09_37x288pt_PanBoreal/user_nl_clm

@@ -0,0 +1,22 @@
+!----------------------------------------------------------------------------------
+! Users should add all user specific namelist changes below in the form of 
+! namelist_var = new_namelist_value 
+!
+! EXCEPTIONS: 
+! Set use_cndv           by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting
+! Set use_vichydro       by the compset you use and the CLM_BLDNML_OPTS -vichydro           setting
+! Set use_cn             by the compset you use and CLM_BLDNML_OPTS -bgc  setting
+! Set use_crop           by the compset you use and CLM_BLDNML_OPTS -crop setting
+! Set spinup_state       by the CLM_BLDNML_OPTS -bgc_spinup      setting
+! Set co2_ppmv           with CCSM_CO2_PPMV                      option
+! Set fatmlndfrc         with LND_DOMAIN_PATH/LND_DOMAIN_FILE    options
+! Set finidat            with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases
+!                        (includes $inst_string for multi-ensemble cases)
+!                        or with CLM_FORCE_COLDSTART to do a cold start
+!                        or set it with an explicit filename here.
+! Set maxpatch_glc       with GLC_NEC                            option
+! Set glc_do_dynglacier  with GLC_TWO_WAY_COUPLING               env variable
+!----------------------------------------------------------------------------------
+
+
+fsurdat = '$CLM_USRDAT_DIR/surfdata_f09_38x288pt_PanBoreal_hist_16pfts_Irrig_CMIP6_simyr2000_c230523.nc'

doc/.ChangeLog_template

@@ -47,6 +47,8 @@ Changes made to namelist defaults (e.g., changed parameter values):
 
 Changes to the datasets (e.g., parameter, surface or initial files):
 
+Changes to documentation:
+
 Substantial timing or memory changes:
 [e.g., check PFS test in the test suite and look at timings, if you
 expect possible significant timing changes]

doc/ChangeLog

@@ -1,4 +1,85 @@
 ===============================================================
+Tag name: ctsm5.2.003
+Originator(s): samrabin (Sam Rabin, UCAR/TSS, samrabin@ucar.edu)
+Date: Thu May  2 14:06:54 MDT 2024
+One-line Summary: Merge b4b-dev
+
+Purpose and description of changes
+----------------------------------
+
+Brings in 4 PRs from b4b-dev to master:
+- Regional CTSM Simulations and Capability of Creating Mesh Files (ESCOMP/CTSM#1892; Negin Sobhani and Adrianna Foster)
+- Add line about documentation in PR template (ESCOMP/CTSM#2488; Sam Rabin)
+- CTSM5.2 2000 fsurdat T42 64x128 file (ESCOMP/CTSM#2495; Sam Levis)
+- Move plumber2 scripts to python directory (ESCOMP/CTSM#2505; Teagan King)
+
+
+Significant changes to scientifically-supported configurations
+--------------------------------------------------------------
+
+Does this tag change answers significantly for any of the following physics configurations?
+(Details of any changes will be given in the "Answer changes" section below.)
+
+    [Put an [X] in the box for any configuration with significant answer changes.]
+
+[ ] clm6_0
+
+[ ] clm5_1
+
+[ ] clm5_0
+
+[ ] ctsm5_0-nwp
+
+[ ] clm4_5
+
+
+Bugs fixed
+----------
+
+List of CTSM issues fixed (include CTSM Issue # and description):
+- Resolves ESCOMP/CTSM#1513: Need a process to subset ESMF mesh files from global ones for regional grids
+- Resolves ESCOMP/CTSM#1773: High resolution regional simulations
+- Resolves ESCOMP/CTSM#2187: Move new PLUMBER2 scripts to python directory to enable python testing
+- Resolves ESCOMP/CTSM#2486: Temporarily add back a T42 dataset for CAM
+
+
+Notes of particular relevance for users
+---------------------------------------
+
+Changes to documentation:
+- Adds documentation for making mesh files
+
+
+Notes of particular relevance for developers:
+---------------------------------------------
+
+Changes to tests or testing:
+- Adds testing for mesh-making Python scripts
+- Adds testing for plumber2_surf_wrapper
+
+
+Testing summary:
+----------------
+
+  regular tests (aux_clm: https://github.com/ESCOMP/CTSM/wiki/System-Testing-Guide#pre-merge-system-testing):
+
+    derecho ----- OK
+    izumi ------- OK
+
+
+Other details
+-------------
+
+Pull Requests that document the changes (include PR ids):
+- ESCOMP/CTSM#2513: Merge b4b-dev 2024-05-02
+- Constituent PRs:
+  - ESCOMP/CTSM#1892: Regional CTSM Simulations and Capability of Creating Mesh Files (https://github.com/ESCOMP/CTSM/pull/1892)
+  - ESCOMP/CTSM#2488: Add line about documentation in PR template (https://github.com/ESCOMP/CTSM/pull/2488)
+  - ESCOMP/CTSM#2495: CTSM5.2 2000 fsurdat T42 64x128 file (https://github.com/ESCOMP/CTSM/pull/2495)
+  - ESCOMP/CTSM#2505: Move plumber2 scripts to python directory (https://github.com/ESCOMP/CTSM/pull/2505)
+
+===============================================================
+===============================================================
 Tag name: ctsm5.2.002
 Originator(s): glemieux (Gregory Lemieux, LBNL, glemieux@lbl.gov)
 Date: Fri 26 Apr 2024 11:13:46 AM MDT

doc/ChangeSum

@@ -1,5 +1,6 @@
 Tag                   Who      Date  Summary
 ============================================================================================================================
+       ctsm5.2.003 samrabin 05/02/2024 Merge b4b-dev
        ctsm5.2.002 glemieux 04/26/2024 FATES default allometry parameter file update
        ctsm5.2.001     erik 04/22/2024 Merge b4b-dev
          ctsm5.2.0     many 04/20/2024 New mksurfdata_esmf tool to create new surface datasets that are in place

doc/design/python_script_user_interface.rst

@@ -81,3 +81,10 @@ More verbose output should go in ``logger.info`` or ``logger.debug`` statements
       * e.g. You might want to output a ``logging.debug`` statement for every variable in a file you are editing.
 
 Near the top of each python module where logging is used, there should be a line, ``logger = logging.getLogger(__name__)``. Then logging statements should be done using statements like ``logger.info(...)``, *not* ``logging.info(...)``: this allows more contextual information in logging output.
+
+====================================================
+ Considerations on inclusion of python packages
+====================================================
+
+Since, this is somewhat an implementation detail the discussion for this is in ``../../python/README.python_pkgs.rst``. The python
+packages used is somewhat both an important part of the user interface, the tool design, and an implementation detail.

doc/source/conf.py

@@ -45,8 +45,8 @@
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
+#source_suffix = '.rst'
 
 # The master toctree document.
 master_doc = 'index'

doc/source/how-to-make-mesh.md

@@ -0,0 +1,98 @@
+# Creating an ESMF mesh file from a netCDF file
+
+This gist includes instructions for creating and visualizing a mesh file from a netcdf file with valid 1D or 2D lats and lons coordinates.
+
+* **ESMF Mesh file** aka **Unstructured Grid File Format** is a netcdf file (format) that includes the information about the grids coordinates and their connectivity to each other. 
+
+Additional information about ESMF mesh files are available [here](https://earthsystemmodeling.org/docs/release/ESMF_8_0_1/ESMF_refdoc/node3.html#SECTION03028200000000000000).
+
+------
+
+In this example, we will use `./mesh_maker.py` which uses `mesh_type.py` to create a mesh file and visualize it.
+
+1- First clone my fork and branch that includes these capabilities:
+``` Shell
+git clone https://github.com/negin513/ctsm.git ctsm_mesh
+cd ctsm_mesh
+
+git checkout subset_mesh_dask
+```
+
+2- Next run mesh_maker.py for a netcdf file:
+
+```
+cd tools/site_and_regional
+```
+Check all the avaialble options:
+
+```
+./mesh_maker.py --help
+```
+
+The output shows all available options for this script:
+```
+|------------------------------------------------------------------|
+|---------------------  Instructions  -----------------------------|
+|------------------------------------------------------------------|
+This script creates ESMF unstructured GRID (mesh file) from a netcdf
+file with valid lats and lons. Provided lats and lons can be 1D or 2D.
+
+For example for running WRF-CTSM cases, the user can create a mesh
+file for their domain :
+    ./mesh_maker.py --input wrfinput_d01 --output my_region
+        --lat XLAT --lon XLONG --verbose
+
+optional arguments:
+  -h, --help        show this help message and exit
+  --input INPUT     Netcdf input file for creating ESMF mesh.
+  --output OUTPUT   Name of the ESMF mesh created.
+  --outdir OUT_DIR  Output directory (only if name of output mesh is not
+                    defined)
+  --lat LAT_NAME    Name of latitude varibale on netcdf input file. If none
+                    given, looks to find variables that include 'lat'.
+  --lon LON_NAME    Name of latitude varibale on netcdf input file. If none
+                    given, looks to find variables that include 'lon'.
+  --mask MASK_NAME  Name of mask varibale on netcdf input file. If none given,
+                    create a fake mask with values of 1.
+  --area AREA_NAME  Name of area variable on netcdf input file. If none given,
+                    ESMF calculates element areas automatically.
+  --overwrite       If meshfile exists, overwrite the meshfile.
+  -v, --verbose     Increase output verbosity
+  
+  ```
+
+Let's create a mesh file from a netcdf file with 1D lats and lons. On the sample files provided 1D lat and long coordinates are saved on `lsmlat` and `lsmlon` variables. 
+
+```
+./mesh_maker.py --input /glade/scratch/negins/example_files/surfdata_4x5_hist_78pfts_CMIP6_simyr1850_275.0-330.0_-40-15_c220705.nc --output test_mesh_1d.nc --lat lsmlat --lon lsmlon --overwrite
+```
+`--verbose` option also provide additional information for debugging. 
+
+This script will create regional and global mesh plots. For example for the above files, the plos are:
+test_mesh_1d_regional.png
+![image](https://user-images.githubusercontent.com/17344536/200441736-972a8136-5c05-4bc9-9bca-b498d972914a.png)
+
+
+test_mesh_1d_global.png
+
+![image](https://user-images.githubusercontent.com/17344536/200441753-d06e95d1-d85b-4216-9c23-d11ba89a31e4.png)
+
+
+
+------
+  ## Creating Mesh files for a WRF domain:
+For running WRF-CTSM cases, we need to create ESMF mesh files for the WRF domain. We can create mesh file from wrfinput (wrf initial condition files). wrfinput has 2D coordinate information on `XLAT` and `XLONG` variable.
+
+For example, let's create a mesh file from a WRF input file for WRF-CTSM run.  
+ ```
+./mesh_maker.py --input /glade/scratch/negins/example_files/wrfinput_d01  --output test_mesh_wrf.nc --lat XLAT --lon XLONG --overwrite
+```
+
+This produce mesh files for running for our WRF domain. 
+
+Here is how the regional plot looks like for this mesh file:
+
+ ![image](https://user-images.githubusercontent.com/17344536/200442002-1ee5595c-9252-4934-a07c-2f6ad86aff1b.png)
+
+
+

py_env_create

@@ -20,6 +20,7 @@ if [ $error != 0 ]; then
    echo "For notes on installing on a user system see: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html"
    echo "Error code was $error"
    cat condahelp.txt
+   rm condahelp.txt
    exit -1
 fi
 rm condahelp.txt
@@ -44,7 +45,7 @@ usage() {
   echo "[-v|--verbose]  "
   echo "     Run with verbose mode for the install so you see the progress bar"
   echo "[-f|--file <file>]  "
-  echo "     Conda environment file to use (can be a text format or YAML format)"
+  echo "     Conda environment requirements text file to use (text format) in addition to the others"
   echo "     Assumed to be under the directory: $condadir"
   echo "     Default is: $condafile"
   echo "[--option <option>]  "
@@ -80,6 +81,9 @@ while [ $# -gt 0 ]; do
    shift
 done
 
+#
+# Error checking options and setup
+#
 if [ ! -f $condadir/$condafile ]; then
    echo "$condadir/$condafile does NOT exist"
    echo "Use the --file option with a valid filename"
@@ -89,11 +93,13 @@ fi
 echo "Use conda to install the python environment needed to run the CTSM python tools in the conda environment: $ctsm_python"
 echo "Using the file: $condadir/$condafile"
 
+#
 # Check if the environment already exists, if it does continue, if not create it
+#
 conda list -n $ctsm_python >& /dev/null
 if [ $? != 0 ]; then
    echo "Create $ctsm_python"
-   cmd="conda create --force -n $ctsm_python -q"
+   cmd="conda create --force --name $ctsm_python --quiet"
    echo "$cmd"
    $cmd
    if [ $? != 0 ]; then
@@ -103,12 +109,20 @@ if [ $? != 0 ]; then
 else
    echo "$ctsm_python environment already exists"
 fi
-echo "Install $ctsm_python this can take a long time, be patient...."
-verbosity="-q"
+#
+# Install the environemnt
+#
+echo "Install $ctsm_python this can take a long time (12 to 20 minutes is expected), be patient...."
+echo "    ...."
+echo "    ...."
+echo "    ...."
+verbosity="--quiet"
+loglevel="ERROR"
 if [ "$verbose" == "Yes" ]; then
-  verbosity="-v"
+  verbosity="--verbose"
+  loglevel="INFO"
 fi
-cmd="conda install --yes $verbosity -c conda-forge -n $ctsm_python --file $condadir/$condafile $option"
+cmd="conda install --yes $verbosity --channel conda-forge --name $ctsm_python $condadir/$condafile $option"
 echo "$cmd"
 $cmd
 if [ $? != 0 ]; then
@@ -117,6 +131,9 @@ if [ $? != 0 ]; then
    echo "Change the file and try again"
    exit -2
 fi
+#
+# Report on success
+#
 echo "Successfully installed the $ctsm_python python environment"
 echo
 echo "activate the environment by doing the following..."

python/README.python_pkgs.rst

@@ -0,0 +1,33 @@
+.. sectnum::
+
+.. contents::
+
+=====================================================================================
+ Requirements to consider for python packages to be included for CTSM python tools
+=====================================================================================
+
+Requirements for CTSM python tools:
+
+Any added dependencies should be discussed and approved by the CTSM-software team. Criteria for evaluation of third-party packages
+include:
+- How much value does this package provide to us beyond what we could get without it?
+- How difficult it is to install this package (including: how much does this complicate / slow down the creation of the conda
+environment)?
+- How stable is this package?
+- How well maintained is this package?
+- Are there other packages that are more stable or better maintained that would provide nearly the same level of value?
+- Tools that require extra packages should be done in a "contrib" type area out of the main part of tools (this would apply for
+advanced plotting capability for example)
+- We need to be able to reproduce working conda environments minimally on our test machines (currently cheyenne and izumi), but also
+on any machine that we run CTSM on. If there is a machine that we can run CTSM on that we can't build the conda environments or run
+the tools on -- that needs to be fixed.
+- We need to tell the user how long to expect the conda environment to load, and give them options if the conda load is taking too
+long
+- Conda environments need to build robustly even for users who don't have ctsm_pylib loaded in their conda environment
+- Currently dask will NOT be something we require for any of the main CTSM tools
+- Currently we won't use conda-lock
+- We specify the black version exactly so that black will function identically for all users
+- We specify the pylint version exactly because pylint is finicky with version and we need it to work identically for all developers
+- We might remove the need for python packages that aren't providing enough utility
+Packages where specific versions seem to be required will have the version requirements in a least the >= form if not an exact
+version

python/conda_env_ctsm_py.txt

@@ -22,3 +22,5 @@ xesmf
 numba=0.55.2  # Avoid 0.56 until numpy>=1.20.  This is the minimum for xesmf
 pylint=2.8.3
 black=22.3.0  # NOTE: The version here needs to be coordinated with the black github action under ../.github/workflows
+cartopy=0.20.2
+matplotlib=3.3.2