Initial docs updates for GCHP run directory changes in 14.0

- Replace input.geos with geoschem_config.yml
- Update all Getting-Started and User-Guide pages
- Scrub the name GCHPctm from the docs
- Remove job script examples
- Move run directory config and configuration files to User-Guide
- Change version of sphinx_rtd_theme from <1.0 to ==0.5.2
  in order to fix TOC formatting issue resolved in 0.5:
  readthedocs/sphinx_rtd_theme#931
- Specify specific version for all other requirements
- Add output files page
- Move plotting output section to User-Guide
- Simplify User-Guide section names that appear on main page

Signed-off-by: Lizzie Lundgren <elundgren@seas.harvard.edu>

docs/requirements.txt

@@ -7,9 +7,9 @@
 # compatibility with the older ReadTheDocs package, so we need to
 # use a version of jinja2 prior to 3.1.0. (Bob Yantosca, 02 May 2022)
 #
-sphinx<4.0.0
-sphinx_rtd_theme<1.0.0
+sphinx==3.5.4
+sphinx_rtd_theme==0.5.2
 sphinxcontrib-bibtex==2.2.0
 recommonmark
 docutils==0.16
-jinja2<3.1.0
+jinja2==3.0.3

docs/source/bibtex.json

@@ -6,7 +6,8 @@
             "Long_et_al._2015",
             "Eastham_et_al._2018",
             "Zhuang_et_al._2020",
-            "Bindle_et_al._2021"
+            "Bindle_et_al._2021",
+            "Lin_et_al._2021",
         ],
         "stretched-grid": [
             "Bindle_et_al._2021"

docs/source/conf.py

@@ -19,11 +19,11 @@
 # -- Project information -----------------------------------------------------
 
 project = 'GCHP'
-copyright = '2021, GEOS-Chem Support Team'
+copyright = '2022, GEOS-Chem Support Team'
 author = 'GEOS-Chem Support Team'
 
 # The full version, including alpha/beta/rc tags
-release = '13.0.2'
+release = '14.0.0'
 
 
 # -- General configuration ---------------------------------------------------

docs/source/getting-started/key-references.rst

@@ -2,7 +2,7 @@ Key References
 ==============
 
 * GEOS-Chem was first described in :cite:`Bey_et_al._2001`. 
-* HEMCO is described in :cite:`Keller_et_al._2014`.
+* HEMCO is described in :cite:`Keller_et_al._2014` and :cite:`Lin_et_al._2021`.
 * Columnar operators are described in :cite:`Long_et_al._2015`.
 * GEOS-Chem High Performance (GCHP) is described in :cite:`Eastham_et_al._2018`.
 * GCHP execution on the cloud and MPI considerations are described in :cite:`Zhuang_et_al._2020`.

docs/source/getting-started/quick-start.rst

@@ -23,7 +23,7 @@ Checkout the GEOS-Chem version that you want to use:
 
 .. code-block:: console
 
-   gcuser:~/GCHP$ git checkout 13.3.4
+   gcuser:~/GCHP$ git checkout 14.0.0
 
 Initialize and update all the submodules:
 
@@ -46,15 +46,17 @@ To create a run directory, run :file:`./createRunDir.sh` and answer the prompts:
 -----------------------
 
 Create a build directory and :command:`cd` into it. 
-A good name for this directory is :file:`build/`, and a good place to put it is the
-top-level of the source code:
+A good name for this directory is :file:`build/`, and you may put this directory anywhere on your system, such as in the top-level of the source code or in your run directory. 
+The GCHP build directory will require 1.4G of storage space. 
+In this guide we put it in the source code:
 
 .. code-block:: console
 
    gcuser:~/GCHP$ mkdir ~/GCHP/build
    gcuser:~/GCHP$ cd ~/GCHP/build
 
-Initialize your build directory by running :program:`cmake`, passing it the path to your source code:
+Initialize your build directory by running :program:`cmake`, passing it the path to your source code. 
+Make sure you have loaded all libraries required for GCHP prior to this step.
 
 .. code-block:: console
 
@@ -63,28 +65,34 @@ Initialize your build directory by running :program:`cmake`, passing it the path
 Now you can configure :ref:`build options <gchp_build_options>`. 
 These are persistent settings that are saved to your build directory.
 A common build option is :literal:`-DRUNDIR`. 
-This option lets you specify one or more run directories that GCHP is "installed" to when you do :command:`make install`. 
+This option lets you specify one or more run directories that GCHP is "installed" to, meaning where the executable is copied, when you do :command:`make install`. 
 Configure your build so it installs GCHP to the run directory you created in Step 2:
 
 .. code-block:: console
 
    gcuser:~/GCHP/build$ cmake . -DRUNDIR="/path/to/your/run/directory"
 
 .. note::
-   The :literal:`.` in the :program:`cmake` command above is important. It tells CMake that your 
-   current working directory (i.e., :literal:`.`) is your build directory.
+   The :literal:`.` in the :program:`cmake` command above is important. It tells CMake that your current working directory (i.e., :literal:`.`) is your build directory.
+
+If you decide instead to build GCHP in your run directory you can do all of the above in one step making use of the :literal:`CodeDir` symbolic link in the run directory:
+
+.. code-block:: console
+
+   gcuser:/path/to/your/run/directory/build$ cmake ../CodeDir -DRUNDIR=..
+
 
 4. Compile and install
 ----------------------
 
-Compiling GCHP takes about 20 minutes, but it can varry depending on your system. 
-Next, compile GCHP:
+Compiling GCHP takes about 20 minutes, but it can vary depending on your system. 
+Next, compile GCHP in parallel using as many cores as are available:
 
 .. code-block:: console
 
    gcuser:~/GCHP/build$ make -j
 
-Next, install the compiled executable to your run directory (or directories):
+Upon successful compilation, install the compiled executable to your run directory (or directories):
 
 .. code-block:: console
 
@@ -96,9 +104,8 @@ This copies :file:`bin/gchp` and supplemental files to your run directory.
    You can update build settings at any time:
 
    1. Navigate to your build directory.
-   2. Update your build settings with :program:`cmake`. See 
-   3. Recompile with :command:`make -j`. Note that the build system automatically figures out what (if any) files
-      need to be recompiled.
+   2. Update your build settings with :program:`cmake`.
+   3. Recompile with :command:`make -j`. Note that the build system automatically figures out what (if any) files need to be recompiled.
    4. Install the rebuilt executable with :command:`make install`.
 
 
@@ -111,48 +118,57 @@ Now, navigate to your run directory:
 
    $ cd path/to/your/run/directory
 
-Most simulation settings are set in :file:`./runConfig.sh`. 
+Commonly changed simulation settings, such as grid resolution, run duration, and number of cores, are set in :file:`setCommonRunSettings.sh`. 
 You should review this file as it explains most settings.
-Note that :file:`./runConfig.sh` is actually a helper script that updates other configuration files. 
-Therefore, you need to run it to actually apply the updates:
+Note that :file:`setCommonRunSettings.sh` is actually a helper script that updates other configuration files. 
+You therefore need to run it to actually apply the settings:
 
 .. code-block:: console
 
-   $ vim runConfig.sh               # edit simulation settings here
-   $ ./runConfig.sh                 # applies the updated settings
+   $ vim setCommonRunSettings.sh           # edit simulation settings here
+   $ ./setCommonRunSettings.sh             # applies the updated settings
+
+Simulation start date is set in :file:`cap_restart`. 
+Run directories come with this file filled in based on date of the initial restart file in subdirectory :file:`Restarts`. 
+You can change the start date only if you have a restart file for the new date in :file:`Restarts`. 
+A symbolic link called :file:`gchp_restart.nc4` points to the restart file for the date in :file:`cap_restart` and the grid resolution in :file:`setCommonRunSettings.sh`.  
+You need to set this symbolic link before running:
+
+.. code-block:: console
+
+   $ ./setRestartLink.sh                   # sets symbolic link to target file in Restarts
+
+If you used an environment file to load libraries prior to building GCHP then you should load that file prior to running. A simple way to make sure you always use the correct combination of libraries is to set the GCHP environment symbolic link :file:`gchp.env` in the run directory:
+
+.. code-block:: console
+
+   $ ./setEnvironment.sh /path/to/env/file # sets symbolic link gchp.env
+   $ source gchp.env                       # applies the environment settings
+
 
 6. Run GCHP
 -----------
 
-Running GCHP is slightly different depending on your MPI library (e.g., OpenMPI, Intel MPI,
-MVAPICH2, etc.) and scheduler (e.g., SLURM, LSF, etc.). If you aren't familiar with running MPI
-programs on your system, see :ref:`Running GCHP <running_gchp>` in the user guide, or ask your
-system administrator.
+Running GCHP is slightly different depending on your MPI library (e.g., OpenMPI, Intel MPI, MVAPICH2, etc.) and scheduler (e.g., SLURM, LSF, etc.). 
+If you aren't familiar with running MPI programs on your system, see :ref:`Running GCHP <running_gchp>` in the user guide, or ask your system administrator.
 
 Your MPI library and scheduler will have a command for launching MPI programs---it's usually something like :program:`mpirun`, :program:`mpiexec`, or :program:`srun`. 
 This is the command that you will use to launch the :program:`gchp` executable.
-You'll have to refer to your system's documentation for specific instructions on running MPI programs,
-but generally it looks something like this:
+You'll have to refer to your system's documentation for specific instructions on running MPI programs, but generally it looks something like this:
 
 .. code-block:: console
 
    $ mpirun -np 6 ./gchp   # example of running GCHP with 6 slots with OpenMPI 
 
-It's recommended you run GCHP as a batch job. This means that you write a (bash) script that runs your 
-GCHP simulation, and then you submit that script to your scheduler (SLURM, LSF, etc.).
+It's recommended you run GCHP as a batch job. 
+This means that you write a script (usually bash) that configures and runs your GCHP simulation, and then you submit that script to your local job scheduler (SLURM, LSF, etc.). 
+Example job scripts are provided in subdirectory :literal:`./runScriptSamples` in the run directory. 
+That folder also includes an example script for running GCHP from the command line.
 
-.. note::
-   When GCHP runs, partially or to completion, it generates several files including
-   :file:`cap_restart` and :file:`gcchem_internal_checkpoint`. Subsequent runs won't
-   overwrite these files, and instead the run will exit with an error. Because of this it is
-   common to do
-
-   .. code-block:: console
-
-      $ rm -f cap_restart gcchem_internal_checkpoint
-
-   before starting a GCHP simulation.
+Several steps beyond running GCHP are included in the example run scripts. These include loading the environment, updating commonly changed run settings, and setting the restart file based on start time and grid resolution.  In addition, the output restart file is moved to the :file:`Restarts` subdirectory and renamed to include start date and grid resolution upon successful completion of the run.
 
+.. note::
+   File :file:`cap_restart` is over-written to contain the run end date upon successful completion of a GCHP run. This is done within GCHP and not by the run script. You can then easily submit a new GCHP run starting off where your last run left off. In addition, GCHP outputs a restart file to your run directory called :file:`gcchem_internal_checkpoint`. This file is moved to subdirectory :literal:`Restarts` and renamed to include the date and grid resolution. This is done by the run script and technically is optional. We recommend doing this since it is is good for archiving (restart files will contain date and grid res) and enables use of the :file:`./setRestartLink.sh` script to set the :file:`gchp_restart.nc4` symbolic link.
 
-Those are the basics of using GCHP! See the user guide, step-by-step guides, and reference pages
-for more detailed instructions.
+Those are the basics of using GCHP! 
+See the user guide, step-by-step guides, and reference pages for more detailed instructions.

docs/source/getting-started/requirements.rst

@@ -6,7 +6,7 @@ System Requirements
 Software Requirements
 ---------------------
 
-To build and run GCHP you compute :term:`environment` needs the following software:
+To build and run GCHP your compute :term:`environment` needs the following software:
 
 * Git
 * Make (or GNUMake)
@@ -57,7 +57,7 @@ Bare Minimum Requirements
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 These bare minimum requirements are sufficient for running GCHP at C24. The are adequate 
-for try GCHP out, and for learning purposes.
+for trying GCHP out, and for learning purposes.
 
 * 6 cores
 * 32 GB of memory
@@ -77,11 +77,21 @@ computational problems:
 
 * Lots of storage. Several TB is sufficient, but tens or hundreds of TB is better.
 
-Other Recommendations
-^^^^^^^^^^^^^^^^^^^^^
+General Hardware and Software Recommendations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 * Hyper-threading may improve simulation throughput, particularly at low core counts
+
 * MPI process should be bound sequentially across cores and nodes (e.g., a simulation with 48-processes with 24 processes per node 
   should bind rank 0 to CPU L#0, rank 1 to CPU L#1, etc. on the first node, and rank 24 to CPU L#0, rank 1 to CPU L#1, etc. on the 
   second node). This should be the default, but it's worth checking if your performance is lower than expected. With OpenMPI the
   `--report-bindings` argument will show you how processes are ranked and binded.
+
+* If using IntelMPI include the following your environment setup to avoid a run-time error:
+
+.. code-block:: bash
+
+    export I_MPI_ADJUST_GATHERV=3
+    export I_MPI_ADJUST_ALLREDUCE=12
+
+* If using OpenMPI and a large number of cores (>1000) we recommend setting :literal:`WRITE_RESTART_BY_OSERVER: YES` in config file :file:`GCHP.rc`. This enables the MAPL o-server functionality for writing restart files, thereby speeding up the mdoel. This is set automatically when executing :file:`setCommonRunSettings.sh`.

docs/source/index.rst

@@ -55,21 +55,20 @@ with Spack, as well as how to use Spack to install GCHP's dependencies if needed
    user-guide/rundir-init.rst
    user-guide/getting-input-data.rst
    user-guide/running.rst
-
+   user-guide/configuration-files.rst
+   user-guide/rundir-config.rst
+   user-guide/output_files.rst
+   user-guide/plotting-output.rst
 
 .. toctree::
    :maxdepth: 1
    :caption: Supplemental Guides
 
-   supplement/rundir-config.rst
-   supplement/config-files.rst
-   supplement/job-script-examples.rst
    supplement/building-gchp-dependencies.rst
    supplement/setting-up-aws-parallelcluster.rst
    supplement/caching-input-data.rst
    supplement/containers.rst
-   supplement/plotting-output.rst
-   stretched-grid.rst
+   supplement/stretched-grid.rst
    supplement/satellite-overpass.rst
    geos-chem-shared-docs/supplemental-guides/bashdatacatalog.rst
    geos-chem-shared-docs/supplemental-guides/using-kpp-with-gc.rst

docs/source/reference/git-submodules.rst

@@ -12,11 +12,11 @@ update submodule so that they use your forks at any time. It is recommended you
 only update the submodules that you need to, and that you leave submodules that
 you don't need to modify pointing to the GEOS-Chem repositories.
 
-The rest of this section assumes you are in the top-level of GCHPctm, i.e.,
+The rest of this section assumes you are in the top-level of GCHP, i.e.,
 
 .. code-block:: console
 
-   $ cd GCHPctm   # navigate to top-level of GCHPctm
+   $ cd GCHP   # navigate to top-level of GCHP
 
 First, identify the submodules that you need to modify. The :file:`.gitmodules`
 file has the paths and URLs to the submodules. You can see it with the following
@@ -72,5 +72,5 @@ Add and commit the update to :file:`.gitmodules`.
    $ git add .gitmodules
    $ git commit -m "Updated submodules to use my own forks"
 
-Now, when you push to your GCHPctm fork, you should see the submodules point to your
-submodule forks.
+Now, when you push to your GCHP fork, you should see the submodules point to your
+submodule forks.

docs/source/supplement/caching-input-data.rst

@@ -140,7 +140,7 @@ Make sure to set :literal:`FIND_PATH` to ExtData and :literal:`REPLACE_PATH` to
    gcuser:/MyRunDirectory$ swap_extdata_link ChemDir
    gcuser:/MyRunDirectory$ swap_extdata_link HcoDir
    gcuser:/MyRunDirectory$ swap_extdata_link MetDir
-   gcuser:/MyRunDirectory$ sed -i "s#${FIND_PATH}#${REPLACE_PATH}#g" HEMCO_Config.rc input.geos
+   gcuser:/MyRunDirectory$ sed -i "s#${FIND_PATH}#${REPLACE_PATH}#g" HEMCO_Config.rc geoschem_config.yml
 
 Now your GCHP simulation will use input data from ExtDataCache.