Skip to content

Commit

Permalink
Initial docs updates for GCHP run directory changes in 14.0
Browse files Browse the repository at this point in the history 
Signed-off-by: Lizzie Lundgren <elundgren@seas.harvard.edu>
  • Loading branch information
@lizziel
lizziel committed Jun 15, 2022
1 parent 2467334 commit f4947d7
Show file tree
Hide file tree
Showing 7 changed files with 143 additions and 139 deletions.
92 changes: 53 additions & 39 deletions docs/source/getting-started/quick-start.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Checkout the GEOS-Chem version that you want to use:

.. code-block:: console
gcuser:~/GCHP$ git checkout 13.3.4
gcuser:~/GCHP$ git checkout 14.0.0
Initialize and update all the submodules:

Expand All @@ -46,15 +46,17 @@ To create a run directory, run :file:`./createRunDir.sh` and answer the prompts:
-----------------------

Create a build directory and :command:`cd` into it.
A good name for this directory is :file:`build/`, and a good place to put it is the
top-level of the source code:
A good name for this directory is :file:`build/`, and you may put this directory anywhere on your system, such as in the top-level of the source code or in your run directory.
The GCHP build directory will require 1.4G of storage space.
In this guide we put it in the source code:

.. code-block:: console
gcuser:~/GCHP$ mkdir ~/GCHP/build
gcuser:~/GCHP$ cd ~/GCHP/build
Initialize your build directory by running :program:`cmake`, passing it the path to your source code:
Initialize your build directory by running :program:`cmake`, passing it the path to your source code.
Make sure you have loaded all libraries required for GCHP prior to this step.

.. code-block:: console
Expand All @@ -63,28 +65,34 @@ Initialize your build directory by running :program:`cmake`, passing it the path
Now you can configure :ref:`build options <gchp_build_options>`.
These are persistent settings that are saved to your build directory.
A common build option is :literal:`-DRUNDIR`.
This option lets you specify one or more run directories that GCHP is "installed" to when you do :command:`make install`.
This option lets you specify one or more run directories that GCHP is "installed" to, meaning where the executable is copied, when you do :command:`make install`.
Configure your build so it installs GCHP to the run directory you created in Step 2:

.. code-block:: console
gcuser:~/GCHP/build$ cmake . -DRUNDIR="/path/to/your/run/directory"
.. note::
The :literal:`.` in the :program:`cmake` command above is important. It tells CMake that your
current working directory (i.e., :literal:`.`) is your build directory.
The :literal:`.` in the :program:`cmake` command above is important. It tells CMake that your current working directory (i.e., :literal:`.`) is your build directory.

If you decide instead to build GCHP in your run directory you can do all of the above in one step making use of the :literal:`CodeDir` symbolic link in the run directory:

.. code-block:: console
gcuser:/path/to/your/run/directory/build$ cmake ../CodeDir -DRUNDIR=..
4. Compile and install
----------------------

Compiling GCHP takes about 20 minutes, but it can varry depending on your system.
Next, compile GCHP:
Compiling GCHP takes about 20 minutes, but it can vary depending on your system.
Next, compile GCHP in parallel using as many cores as are available:

.. code-block:: console
gcuser:~/GCHP/build$ make -j
Next, install the compiled executable to your run directory (or directories):
Upon successful compilation, install the compiled executable to your run directory (or directories):

.. code-block:: console
Expand All @@ -96,9 +104,8 @@ This copies :file:`bin/gchp` and supplemental files to your run directory.
You can update build settings at any time:

1. Navigate to your build directory.
2. Update your build settings with :program:`cmake`. See
3. Recompile with :command:`make -j`. Note that the build system automatically figures out what (if any) files
need to be recompiled.
2. Update your build settings with :program:`cmake`.
3. Recompile with :command:`make -j`. Note that the build system automatically figures out what (if any) files need to be recompiled.
4. Install the rebuilt executable with :command:`make install`.


Expand All @@ -111,48 +118,55 @@ Now, navigate to your run directory:
$ cd path/to/your/run/directory
Most simulation settings are set in :file:`./runConfig.sh`.
Commonly changed simulation settings, such as grid resolution, run duration, and number of cores, are set in :file:`./setCommonRunSettings.sh`.
You should review this file as it explains most settings.
Note that :file:`./runConfig.sh` is actually a helper script that updates other configuration files.
Therefore, you need to run it to actually apply the updates:
Note that :file:`./setCommonRunSettings.sh` is actually a helper script that updates other configuration files.
You therefore need to run it to actually apply the settings:

.. code-block:: console
$ vim setCommonRunSettings.sh # edit simulation settings here
$ ./setCommonRunSettings.sh # applies the updated settings
Simulation start date is set in :file:`./cap_restart`.
Run directories come with this file filled in based on date of the initial restart file in subdirectory :literal:`./Restarts`.
You can change the start date only if you have a restart file for the new date in :literal:`./Restarts`.
A symbolic link called :file:`gchp_restart.nc4` points to the restart file for the date in :file:`./cap_restart` and the grid resolution in :file:`./setCommonRunSettings.sh`.
You need to set this symbolic link before running:

.. code-block:: console
$ vim runConfig.sh # edit simulation settings here
$ ./runConfig.sh # applies the updated settings
$ ./setRestartLink.sh # sets symbolic link gchp_restart.nc4
If you used an environment file to load libraries prior to building GCHP then you should load that file prior to running GCHP. A simple way to make sure you always use the correct combination of libraries is to set the GCHP environment symbolic link :literal:`./gchp.env` in the run directory:

$ ./setEnvironment.sh /path/to/env/file # sets symbolic link gchp.env
$ source gchp.env


6. Run GCHP
-----------

Running GCHP is slightly different depending on your MPI library (e.g., OpenMPI, Intel MPI,
MVAPICH2, etc.) and scheduler (e.g., SLURM, LSF, etc.). If you aren't familiar with running MPI
programs on your system, see :ref:`Running GCHP <running_gchp>` in the user guide, or ask your
system administrator.
Running GCHP is slightly different depending on your MPI library (e.g., OpenMPI, Intel MPI, MVAPICH2, etc.) and scheduler (e.g., SLURM, LSF, etc.).
If you aren't familiar with running MPI programs on your system, see :ref:`Running GCHP <running_gchp>` in the user guide, or ask your system administrator.

Your MPI library and scheduler will have a command for launching MPI programs---it's usually something like :program:`mpirun`, :program:`mpiexec`, or :program:`srun`.
This is the command that you will use to launch the :program:`gchp` executable.
You'll have to refer to your system's documentation for specific instructions on running MPI programs,
but generally it looks something like this:
You'll have to refer to your system's documentation for specific instructions on running MPI programs, but generally it looks something like this:

.. code-block:: console
$ mpirun -np 6 ./gchp # example of running GCHP with 6 slots with OpenMPI
It's recommended you run GCHP as a batch job. This means that you write a (bash) script that runs your
GCHP simulation, and then you submit that script to your scheduler (SLURM, LSF, etc.).

.. note::
When GCHP runs, partially or to completion, it generates several files including
:file:`cap_restart` and :file:`gcchem_internal_checkpoint`. Subsequent runs won't
overwrite these files, and instead the run will exit with an error. Because of this it is
common to do

.. code-block:: console
It's recommended you run GCHP as a batch job.
This means that you write a script (usually bash) that configures and runs your GCHP simulation, and then you submit that script to your local job scheduler (SLURM, LSF, etc.).
Example job scripts are provided in subdirectory :literal:`./runScriptSamples` in the run directory.
That folder also includes an example script for running GCHP from the command line.

$ rm -f cap_restart gcchem_internal_checkpoint
before starting a GCHP simulation.
Several steps beyond running GCHP are included in the example run scripts. These include loading the environment, updating common settings, and setting the restart file link based on start time and grid resolution. In addition, the output restart file is moved and renamed to the :literal:`./Restarts` subdirectory upon successful completion of the run.

.. note::
When GCHP runs to completion it updates the start date in :file:`cap_restart` to be the end date of your run. This is so that you can easily submit a new GCHP run starting off where your last run left off. If your GCHP run failed to complete this file will not be updated. In addition, GCHP outputs a restart file to your run directory called :file:`gcchem_internal_checkpoint`. Upon successful completion of a run this file should be moved and renamed to subdirectory :literal:`./Restarts` to include the date and grid resolution. This is good for archiving and enables the :file:`./setRestartLink.sh` script to work for the new start time. This step is included in all sample GCHP run scripts.

Those are the basics of using GCHP! See the user guide, step-by-step guides, and reference pages
for more detailed instructions.
Those are the basics of using GCHP!
See the user guide, step-by-step guides, and reference pages for more detailed instructions.
90 changes: 40 additions & 50 deletions docs/source/stretched-grid.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,14 @@ Stretched-Grid Simulations
you should consider, and offers guidance for choosing appropriate stretching parameters.

A stretched-grid is a cubed-sphere grid that is "stretched" to enhance its resolution in a region.
To set up a stretched-grid simulation you need to do two things:
To set up a stretched-grid simulation you need to do the following:

#. Create a restart file for your simulation.
#. Update :file:`runConfig.sh` to specify the grid and restart file.
#. Choose stretching parameters, including stretch factor and target latitude and longitude.
#. Create a stretched grid restart file for your simulation using your chosen stretch parameters.
#. Update :file:`setCommonRunSettings.sh` to specify the stretched grid used in the restart file.
#. Configure using the stretched grid restart file

Before setting up your stretched-grid simulation, you will need to choose stretching parameters.
Before starting you will need to choose stretching parameters.

Choose stretching parameters
----------------------------
Expand Down Expand Up @@ -63,19 +65,19 @@ GCPy documentation for this program's exact usage, and for installation instruct

.. code-block:: console
$ python -m gcpy.file_regrid \
-i initial_GEOSChem_rst.c90_standard.nc \
--dim_format_in checkpoint \
-o sg_restart_c48_3_260_40.nc \
--cs_res_out 48 \
--sg_params_out 3.0 260.0 40.0 \
$ python -m gcpy.file_regrid \
-i GEOSChem.Restart.20190701_000000z.c90.nc4 \
--dim_format_in checkpoint \
-o sg_restart_c48_3_260_40.nc \
--cs_res_out 48 \
--sg_params_out 3.0 260.0 40.0 \
--dim_format_out checkpoint
Description of arguments:

.. option:: -i initial_GEOSChem_rst.c90_standard.nc
.. option:: -i initial_GEOSChem.Restart.20190701_000000z.c90.nc

Specifies the input restart file is :file:`initial_GEOSChem_rst.c90_standard.nc` (in the current working directory).
Specifies the input restart file is :file:`GEOSChem.Restart.20190701_000000z.c90.nc4` (in the current working directory).


.. option:: --dim_format_in checkpoint
Expand Down Expand Up @@ -104,61 +106,41 @@ simulation's configuration files.
Update your configuration files
-------------------------------

Modify the section of :file:`runConfig.sh` that controls the simulation grid. Turn
Modify the section of :file:`setCommonRunSettings.sh` that controls the simulation grid. Turn
:envvar:`STRETCH_GRID` to :literal:`ON` and update :envvar:`CS_RES`, :envvar:`STRETCH_FACTOR`,
:envvar:`TARGET_LAT`, and :envvar:`TARGET_LON` for your specific grid.

.. code-block:: bash
#------------------------------------------------
# Internal Cubed Sphere Resolution
#------------------------------------------------
# Primary resolution is an integer value. Set stretched grid to ON or OFF.
# 24 ~ 4x5, 48 ~ 2x2.25, 90 ~ 1x1.25, 180 ~ 1/2 deg, 360 ~ 1/4 deg
#------------------------------------------------
# GRID RESOLUTION
#------------------------------------------------
# Integer representing number of grid cells per cubed-sphere face side
CS_RES=24
STRETCH_GRID=ON
# Stretched grid parameters
# Rules and notes:
# (1) Minimum STRETCH_FACTOR is 1.0001
# (2) Target lat and lon must be floats (contain decimal)
# (3) Target lon must be in range [0,360)
#------------------------------------------------
# STRETCHED GRID
#------------------------------------------------
# Turn stretched grid ON/OFF. Follow these rules if ON:
# (1) Minimum STRETCH_FACTOR value is 1.0001
# (2) TARGET_LAT and TARGET_LON are floats containing decimal
# (3) TARGET_LON in range [0,360)
STRETCH_GRID=OFF
STRETCH_FACTOR=3.0
TARGET_LAT=40.0
TARGET_LON=260.0
Next, modify the section of :file:`runConfig.sh` that specifies the simulation restart file.
Set :envvar:`INITIAL_RESTART` to the restart file we created in the :ref:`previous step <sg_restart_file_regridding>`.

.. code-block:: bash
#------------------------------------------------
# Initial Restart File
#------------------------------------------------
# By default the linked restart files in the run directories will be
# used. Please note that HEMCO restart variables are stored in the same
# restart file as species concentrations. Initial restart files available
# on gcgrid do not contain HEMCO variables which will have the same effect
# as turning the HEMCO restart file option off in GC classic. However, all
# output restart files will contain HEMCO restart variables for your next run.
# INITIAL_RESTART=initial_GEOSChem_rst.c${CS_RES}_TransportTracers.nc
# You can specify a custom initial restart file here to overwrite:
INITIAL_RESTART=sg_restart_c48_3_260_40.nc
Lastly, execute :program:`./runConfig.sh` to update to update your run directory's
configuration files.
Execute :program:`./setCommonRunSettings.sh` to update to update your run directory's configuration files.

.. code-block:: console
$ ./runConfig.sh
$ ./setCommonRunSettings.sh
.. warning::

For stretched-grid simulation with GCHP version 13.4.0 and earlier, the
time steps for chemistry and transport needs to be specified manually. You can do this
by replacing the if/else block that sets the time steps in :file:`runConfig.sh` with
by replacing the if/else block that sets the time steps in :file:`setCommonRunSettings.sh` with

.. code-block::
Expand All @@ -167,4 +149,12 @@ configuration files.
TransConv_Timestep_HHMMSS=000500
if your effective resolution is greater than or equal to C180-equivalent.



Configure the restart file
--------------------------

Update :file:`cap_restart` to match the date of your restart file which will also be the start date of the run.
Copy or symbolically link your restart file in the :literal:`Restarts` subdirectory with the proper filename format. The format includes global resolution but not stretched grid resolution so it is a good idea to symbolically link it if you want to preserve the original file's specification of stretched grid in its name.
Run :literal:`setRestartLink.sh` to set symbolic link :file:`gchp_restart.nc4` to point to your restart file based on start date in :file:`cap_restart` and global grid resolution in :file:`setCommonRunSettings.sh`.

4 changes: 2 additions & 2 deletions docs/source/supplement/config-files.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ These files contain all run-time information required by GCHP. Files include:
config-files/input_geos.rst
config-files/others.rst

Much of the labor of updating the configuration files has been eliminated by run directory shell script :file:`runConfig.sh`.
It is important to remember that sourcing :file:`runConfig.sh` will overwrite settings in other configuration files, and you therefore should never manually update other configuration files unless you know the specific option is not available in :file:`runConfig.sh`.
Much of the labor of updating the configuration files has been eliminated by run directory shell script :file:`setCommonRunSettings.sh`.
It is important to remember that sourcing :file:`setCommonRunSettings.sh` will overwrite settings in other configuration files, and you therefore should never manually update other configuration files unless you know the specific option is not available in :file:`setCommonRunSettings.sh`.

That being said, it is still worth understanding the contents of all configuration files and what all run options include.
This page details the settings within all configuration files and what they are used for.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/supplement/containers.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Replace the typical execution line in the script (where ``mpirun`` or ``srun`` i
$ time mpirun singularity exec ../gchp.sif /rundir/internal_exec >> ${log}
You can now setup your run configuration as normal using `runConfig.sh` and tweak Slurm parameters in your run script.
You can now setup your run configuration as normal using `setCommonRunSettings.sh` and tweak Slurm parameters in your run script.


If you already have GCHP data directories, congratulations! You've completed all the steps you need to run GCHP in a container.
Expand Down

0 comments on commit f4947d7

Please sign in to comment.