Merge develop into BRIDGE in prep for release v18.0.1 (#753)

README.md

@@ -2,6 +2,11 @@
 
 This document explains how to build, set up, and run the GEOS land modeling and data assimilation system (`GEOSldas`) on the most common systems used by GMAO.  Additional steps are needed on other systems.
 
+## Note: GEOSldas Repository Structure Revised (March 2024)
+
+On 26 March 2024, GEOSldas was restructured and split into two repositories.  Specifically, the GEOSldas source code was moved from the GEOSldas fixture into a new repository, the [GEOSldas_GridComp](https://github.com/GEOS-ESM/GEOSldas_GridComp), which is now imported into the fixture as an external repository.  Note that the [GEOSldas_GridComp](https://github.com/GEOS-ESM/GEOSldas_GridComp) includes the "Applications" directory [`./GEOSldas_App`](https://github.com/GEOS-ESM/GEOSldas_GridComp/tree/develop/GEOSldas_App).
+
+
 ## How to Build GEOSldas
 
 ### Step 1: Load the Build Modules

components.yaml

@@ -46,8 +46,14 @@ MAPL:
   tag: v2.44.1
   develop: develop
 
+GEOSldas_GridComp:
+  local: ./src/Components/@GEOSldas_GridComp
+  remote: ../GEOSldas_GridComp.git
+  branch: develop
+  develop: develop
+
 GEOSgcm_GridComp:
-  local: ./src/Components/GEOSldas_GridComp/@GEOSgcm_GridComp
+  local: ./src/Components/@GEOSldas_GridComp/@GEOSgcm_GridComp
   remote: ../GEOSgcm_GridComp.git
   branch: develop
   sparse: ./config/GEOSgcm_GridComp_ldas.sparse

doc/CHANGELOG.md

@@ -27,12 +27,25 @@ this period, LDASsa and GEOSldas development continued in parallel.
 
 In 2019, GEOS LDAS version control transferred from CVS to Git.
 
+In March 2024, GEOSldas was split into two repositories, the GEOSldas fixture and the GEOSldas_GridComp.
+
 This README file contains the history of stable GEOSldas Releases in Git, followed by older, CVS LDASsa tags and GEOSldas versions and change logs.
 
 
 Overview of Git Releases:
 ============================
 
+[v18.0.1](https://github.com/GEOS-ESM/GEOSldas/releases/tag/v18.0.1) - 2024-03-27
+------------------------------
+
+- 0-diff vs. v18.0.0.
+- Essentially identical to v18.0.0 except for revised repository structure after split into two repositories: [GEOSldas](https://github.com/GEOS-ESM/GEOSldas) and [GEOSldas_GridComp](https://github.com/GEOS-ESM/GEOSldas_GridComp) ([PR #748](https://github.com/GEOS-ESM/GEOSldas/pull/748), [PR #750](https://github.com/GEOS-ESM/GEOSldas/pull/750)).
+
+- Minor changes:
+  - Support for running "sbatch [FULL_PATH/]lenkf.j" from any directory ([PR #745](https://github.com/GEOS-ESM/GEOSldas/pull/745)).
+
+
+------------------------------
 [v18.0.0](https://github.com/GEOS-ESM/GEOSldas/releases/tag/v18.0.0) - 2024-03-22
 ------------------------------
 

doc/README.ConfigurationFiles.md

@@ -1,32 +1,32 @@
 # GEOSldas Configuration Files
 
-This document describes the files involved in the specification and pre-processing of the GEOSldas configuration parameters. 
+This document describes the files involved in the specification and pre-processing of the GEOSldas configuration parameters.
 
 Note that the values of the configuration parameters that are used during the execution of `GEOSldas.x` are written into the GEOSldas log file (`./output/*/rc_out/Y*/M*/*.ldas_log.*.txt`).
 
 ---
 ### `"exeinp"` and `"batinp"` files
 
-Inputs to `ldas_setup` that contain user-defined configuration information.  
+Inputs to `ldas_setup` that contain user-defined configuration information.
 
-Use `ldas_setup` to create sample files that contain descriptions of  the configuration parameters.  
+Use `ldas_setup` to create sample files that contain descriptions of  the configuration parameters.
 
 For details, see [README.md](https://github.com/GEOS-ESM/GEOSldas/blob/main/README.md).
 
 
 ---
 ### `LDASsa_SPECIAL_inputs_*.nml`
-
-Optional Fortran namelist (nml) files that contain additional user-defined configuration information for ensemble perturbations and data assimilation.  
 
-The path to these nml files is specified in the `"exeinp"` file. 
+Optional Fortran namelist (nml) files that contain additional user-defined configuration information for ensemble perturbations and data assimilation.
 
-During `ldas_setup`, **default** configuration files (`LDASsa_DEFAULT_inputs_*.nml`) are created in the experiment `./run` directory.  These files contain a complete set of the available configuration parameters with descriptions.  The default configuration is a single-member land model simulation without perturbations and without data assimilation. The "DEFAULT" nml files must be present in the experiment `./run` directory and should not be edited.  
+The path to these nml files is specified in the `"exeinp"` file.
+
+During `ldas_setup`, **default** configuration files (`LDASsa_DEFAULT_inputs_*.nml`) are created in the experiment `./run` directory.  These files contain a complete set of the available configuration parameters with descriptions.  The default configuration is a single-member land model simulation without perturbations and without data assimilation. The "DEFAULT" nml files must be present in the experiment `./run` directory and should not be edited.
 
 To run GEOSldas with ensemble perturbations and data assimilation, users must create "SPECIAL" nml files (`LDASsa_SPECIAL_inputs_*.nml`) that contain the desired settings of the parameters.  Only the nml parameters that are different from those in the "DEFAULT" files need to be included in the "SPECIAL" nml files.
 
 Parameters are grouped into three separate nml files:
-* `ensprop` : Perturbations applied during the land model ensemble propagation step. 
+* `ensprop` : Perturbations applied during the land model ensemble propagation step.
 * `ensupd`  : Assimilated observations and parameters of the ensemble-based analysis.
 * `catbias` : Dynamic bias estimation (defunct).
 
@@ -50,11 +50,11 @@ If an experiment did not complete successfully (e.g., because of a system downti
 
 Contains parameters needed to run the main GEOSldas executable (`GEOSldas.x`), including information on land model configuration, version, grid, time steps, processor layout.
 
-Created in the experiment `./run` directory during `ldas_setup`.  Merges information from the `"exeinp"` and `"batinp"` files and the resource parameter template files for GEOSldas, i.e., [GEOSldas_LDAS.rc](https://github.com/GEOS-ESM/GEOSldas/blob/main/src/Applications/LDAS_App/GEOSldas_LDAS.rc) and  [GEOS_SurfaceGridComp.rc](https://github.com/GEOS-ESM/GEOSgcm_GridComp/blob/main/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared/GEOS_SurfaceGridComp.rc) (from the linked GEOSgcm_GridComp repository).
+Created in the experiment `./run` directory during `ldas_setup`.  Merges information from the `"exeinp"` and `"batinp"` files and the resource parameter template files for GEOSldas, i.e., [GEOSldas_LDAS.rc](https://github.com/GEOS-ESM/GEOSldas/blob/main/src/Components/GEOSldas_GridComp/GEOSldas_App/GEOSldas_LDAS.rc) and  [GEOS_SurfaceGridComp.rc](https://github.com/GEOS-ESM/GEOSgcm_GridComp/blob/main/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/Shared/GEOS_SurfaceGridComp.rc) (from the linked GEOSgcm_GridComp repository).
 
 **Users are generally discouraged from editing** `LDAS.rc`.  For example, editing `MET_PATH` does not change the associated directory link.  Instead, users should run `ldas_setup` to create `LDAS.rc`.
 
-`LDAS.rc` can be useful as a compact overview of the experiment configuration. 
+`LDAS.rc` can be useful as a compact overview of the experiment configuration.
 
 ---
 ### `lenkf.j`

doc/README.OutputSpecs.md

@@ -7,26 +7,26 @@ Most diagnostic output from GEOSldas is generated through the **HISTORY** functi
 Restart/checkpoint files are also written through MAPL (although not through HISTORY).  The key configuration parameters that control checkpoint file output are described in the sample "exeinp" files that can be generated using `ldas_setup`.  For details, see [README.md](https://github.com/GEOS-ESM/GEOSldas/blob/main/README.md).
 
 Notable exceptions from the MAPL-generated output include:
-* log files [ASCII], 
+* log files [ASCII],
 * observation-space "ObsFcstAna" data assimilation diagnostics [BINARY], and
 * SMAP L4_SM-specific "aup" data assimilation diagnostics files (available _**only**_ for simulations in EASEv2_M09 tile space) [BINARY].
 
-Output of the latter two sets of files can be turned on/off in the `[NML_INPUT_PATH]/LDASsa_SPECIAL_inputs_ensupd.nml` configuration file, and Matlab readers are available in the 
-[src/Applications/LDAS_App/util](https://github.com/GEOS-ESM/GEOSldas/tree/main/src/Applications/LDAS_App/util) directory.
+Output of the latter two sets of files can be turned on/off in the `[NML_INPUT_PATH]/LDASsa_SPECIAL_inputs_ensupd.nml` configuration file, and Matlab readers are available in the
+[src/Components/GEOSldas_GridComp/GEOSldas_App/util](https://github.com/GEOS-ESM/GEOSldas/tree/main/src/Components/GEOSldas_GridComp/GEOSldas_App/util) directory.
 
 
 ### MAPL HISTORY output
 
-As part of `ldas_setup`, a sample `HISTORY.rc` configuration file is created in the experiment's `./run` directory.  Users specify the desired output by editing `HISTORY.rc`.  
+As part of `ldas_setup`, a sample `HISTORY.rc` configuration file is created in the experiment's `./run` directory.  Users specify the desired output by editing `HISTORY.rc`.
 
 `HISTORY.rc` defines a number of output file "Collections", each of which contains one or more output variables.  Output can be in the native tile space ("1d") or gridded ("2d"), _**except**_ when the simulation is in the EASE grid tile space (see below).
 
 All variables contained in a given Collection are written:
-* on the same ("2d") grid (if gridded), 
-* at the same frequency, and 
-* with either time-average ("tavg") or instantaneous ("inst") sampling mode.  
+* on the same ("2d") grid (if gridded),
+* at the same frequency, and
+* with either time-average ("tavg") or instantaneous ("inst") sampling mode.
 
-In the following example, two Collections are written.  The `tavg3_2d_lnd_Nx` Collection contains time-average ("tavg"), 3-hourly ("3"), gridded ("2d") data, and the `inst1_1d_lfs_Nt` Collection contains snapshot/instantaneous ("inst"), 1-hourly, tile-space ("1d") data. 
+In the following example, two Collections are written.  The `tavg3_2d_lnd_Nx` Collection contains time-average ("tavg"), 3-hourly ("3"), gridded ("2d") data, and the `inst1_1d_lfs_Nt` Collection contains snapshot/instantaneous ("inst"), 1-hourly, tile-space ("1d") data.
 ```
 COLLECTIONS:
             'tavg3_2d_lnd_Nx'
@@ -46,12 +46,12 @@ For example, to write 3-hourly, time-average output of the "WCSF" and "WCRZ" var
                               ::
 ```
 
-To be available for output through MAPL HISTORY, a variable ("field") must be defined as an `ExportSpec` in a `GEOS_*GridComp.F90` file.  The list of variables ("fields") in the definition of each Collection consists of three columns: 
+To be available for output through MAPL HISTORY, a variable ("field") must be defined as an `ExportSpec` in a `GEOS_*GridComp.F90` file.  The list of variables ("fields") in the definition of each Collection consists of three columns:
 - (column 1) variable name in `GEOS_[GCNAME]GridComp.F90` file,
-- (column 2) GridComp name [GCNAME], and 
+- (column 2) GridComp name [GCNAME], and
 - (column 3) user-specified variable name that appears in nc4 output (optional).
 
-The same variable can be written in more than one Collection (i.e., at the same or different temporal and/or spatial resolutions), and there is no limit to the number of Collections that are defined for and written by a simulation.  
+The same variable can be written in more than one Collection (i.e., at the same or different temporal and/or spatial resolutions), and there is no limit to the number of Collections that are defined for and written by a simulation.
 
 Gridded ("2d") output can be on a grid other than the "native" grid that is associated with the tile space used in the simulation, as long as the output grid is defined in `HISTORY.rc`.  For example, if the simulation uses a cube-sphere tile space, the 1/2-degree lat/lon output grid mentioned above would be defined as follows:
 ```
@@ -71,20 +71,20 @@ MAPL HISTORY can generally write gridded ("2d") output in binary or netcdf-4 (nc
 
 **Special considerations for GEOSldas**
 
-1. Gridded ("2d") output _**cannot**_ be written for simulations in any EASE-grid tile space.  But since each EASE-grid cell contains at most one tile, it is straightforward to convert tile-space ("1d") output into gridded output (2d arrays) on the fly using the `i_indg` and `j_indg` data in the `./rc_out/*tilecoord*` file.  See Matlab readers and scripts in `./src/Application/LDAS_App/util`.
+1. Gridded ("2d") output _**cannot**_ be written for simulations in any EASE-grid tile space.  But since each EASE-grid cell contains at most one tile, it is straightforward to convert tile-space ("1d") output into gridded output (2d arrays) on the fly using the `i_indg` and `j_indg` data in the `./rc_out/*tilecoord*` file.  See Matlab readers and scripts in `./src/Components/GEOSldas_GridComp/GEOSldas_App/util`.
 
-2. When running in EASE-grid tile space, the main GEOSldas executable (`GEOSldas.x`) first writes tile-space ("1d") output in binary format using MAPL HISTORY.  If nc4 output is requested in `HISTORY.rc`, the binary output from `GEOSldas.x` is then automatically converted into nc4 format in post-processing as part of the `lenkf.j` job script using the `tile_bin2nc4.F90` utility. 
+2. When running in EASE-grid tile space, the main GEOSldas executable (`GEOSldas.x`) first writes tile-space ("1d") output in binary format using MAPL HISTORY.  If nc4 output is requested in `HISTORY.rc`, the binary output from `GEOSldas.x` is then automatically converted into nc4 format in post-processing as part of the `lenkf.j` job script using the `tile_bin2nc4.F90` utility.
 
-3. GEOSldas can bundle sub-daily nc4 output into daily nc4 files and write monthly-average output through the `POSTPROC_HIST` configuration option. 
+3. GEOSldas can bundle sub-daily nc4 output into daily nc4 files and write monthly-average output through the `POSTPROC_HIST` configuration option.
 
 
 **Enhanced file compression and bit shaving**
 
-To save disk space, MAPL can facilitate enhanced file compression through the modification of scientifically meaningless information in the output files. The `\*.nbits` parameter specifies the number of bits retained: 
+To save disk space, MAPL can facilitate enhanced file compression through the modification of scientifically meaningless information in the output files. The `\*.nbits` parameter specifies the number of bits retained:
 ```
  tavg3_2d_lnd_Nx.nbits:        12,
 ```
-Many MERRA-2 and FP products, for example, use `\*.nbits: 12` and `\*.nbits: 10`, respectively.  
+Many MERRA-2 and FP products, for example, use `\*.nbits: 12` and `\*.nbits: 10`, respectively.
 
 To realize the disk space savings, bit-shaved output **must be compressed separately** after the simulation has finished.   Binary files can be compressed with `gzip`; nc4 files can be compressed using the `compress_bit-shaved_nc4.sh` utility script. For reasons of efficiency, the compression is not included in GEOSldas `POSTPROC_HIST`.