[PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic #459

LiamBindle · 2020-09-25T19:03:01Z

Hi everyone,

This PR contains updates to run directory creation to facilitate automated run directory creation (and in general, run directory creation from a list of files with variables describing the state to initialize the run directory to). This PR is zero-diff on run directories created with ./createRunDir.sh, but it does change the underlying way in which ./createRunDir.sh works.

In essence, ./createRunDir.sh used to run a bunch of sed commands to tokens in the templates. This PR replaces this sed-based approach with the envsubst-based approach, and moves the copying/templating of required files to a new script called init_rd.sh. This new script has the following usage:

$ init_rd.sh -h
Initialize the current working directory as a run directory (minimal file set). 

usage: 
   init_rd.sh FILE...
      FILE...: Files defining variables used to initialize run directory. The
               rightmost file has the highest precedence.
               
               Each FILE is sourced and the loaded variables are used to
               fill the run directory templates using envsubst.

   init_rd.sh --rdi-vars
      Lists all the RDI variables found in the templates.

   init_rd.sh [-h] [--help]
      Prints this help message.

In summary, init_rd.sh takes a list of files, loads the variables in those files, and then fills the templates (initializing the cwd as the run directory).

This PR updates ./createRunDir.sh to use init_rd.sh. Note that ./createRunDir.sh is still used to create all the supplemental files (e.g., runSampleScripts, etc.).

Additional Notes

The variables that used to initialize run directory settings used to be quite vague (understandably since they were only used internally in ./createRunDir.sh). In this PR I've updated the variables to be more descriptive and prefixed with RDI_ (standing for Run Directory Initialization). This has the added benefit of allowing the --rdi-vars options which lists all the RDI variables found in the templates.

The main feature of this PR is init_rd.sh. I didn't intend for this script to be used by the public, but I do think it's useful to advanced users. It allows run directories to be created rapidly, precisely, and without the interactive wizard. This facilitates automated run directory creation.

You can add the directory containing init_rd.sh to your $PATH. In doing this, you can run init_rd.sh from anywhere, and it will resolve the template locations correctly.

Example

This example demonstrates creating a benchmark run directory (in a way that could be done by an automated system). First, we need a file with all the RDI variables:

RDI_DATA_ROOT=/ExtData
RDI_SIM_NAME=fullchem
RDI_MET="MERRA2"
RDI_MET_SOURCE=$RDI_MET

RDI_MET_RES="05x0625"
RDI_MET_NATIVE_RES="0.5x0.625"
RDI_MET_LAT_RES="05"
RDI_MET_LON_RES="0625"
RDI_MET_EXT="nc4"

RDI_MET_CN_YR=2015
RDI_MET_PRES_UNIT="Pa "
RDI_MET_PRES_SCALE="0.01"
RDI_MET_DUST_SF="3.86e-4"
RDI_MET_DIR=$RDI_DATA_ROOT/GEOS_0.5x0.625/MERRA2
RDI_RESTART_FILE='initial_GEOSChem_rst.c${CS_RES}'_fullchem.nc
RDI_HIST_TIME_AVG_DUR="7440000"
RDI_HIST_TIME_AVG_FREQ="7440000"
RDI_HIST_INST_DUR="7440000"
RDI_HIST_INST_FREQ="7440000"
RDI_HIST_MONTHLY_DIAG="1"

RDI_CS_RES="48"

RDI_NUM_CORES="48"
RDI_CORES_PER_NODE="24"
RDI_NUM_NODES="2"

RDI_SIM_START_DATE="20190701"
RDI_SIM_END_DATE="20190801"
RDI_SIM_START_TIME="000000"
RDI_SIM_END_TIME="000000"
RDI_SIM_DUR_YYYYMMDD="00000100"
RDI_SIM_DUR_HHmmSS="000000"

For this example, pretend this file is called benchmark_rdi_vars.txt. Then, I can create a benchmark run directory like so

$ mkdir benchmark-rundir
$ cd benchmark-rundir
$ init_rd.sh path/to/benchmark_rdi_vars.txt

This initializes the cwd as the run directory. Note that the RDI variables can be split across multiple files. Also note that RDI variables can be specified multiple times, in which case the rightmost file has the highest precedence.

The resulting run directory looks like so

$ ls
CAP.rc	ChemDir  ExtData.rc  GCHP.rc  HcoDir  HEMCO_Config.rc  HEMCO_Diagn.rc  HISTORY.rc  input.geos  input.nml  MetDir  OutputDir  runConfig.sh

Co-benefits

This also make run directory creation more modular. For example, RDI variables for simulation type {fullchem, TransportTracer, benchmark} are split into their own .txt files in the new settings/ subdirectory, and the RDI variables for {MERRA2, GEOS-FP} are also in their own .txt files. This should ease run directory creation maintanance.

Zero-diff tests

Check marks indicate tests that passed.

Simulation types:

Full chemistry sim
TransportTracer

Simulation options:

Meteorology source:

MERRA2
GEOS-FP

Since each of the three categories above control settings that are mutually exclusive, a single test can fulfill multiple
categories (i.e. each one needs to be tested at least once, but combinations don't need to be checked).

Additional tests

Now, when you run ./createRunDir.sh it will create a file named rdi_vars.sh in the run directory. This file contains
the Run Directory Initialization variables that were used to create the run directory. Therefore,
the essential run directory files must be reproducable with the following command:

envsubst_create.sh rdi_vars.txt

This test passed (i.e. all the essential run directory files were reproduced identically; checked with diff).

LiamBindle · 2020-09-25T19:04:12Z

Oops, I didn't realize that alpha.10 was behind in the run directory. I'll work on rebasing on dev/13.0.0.

lizziel · 2020-09-25T19:11:17Z

Is this needed for 13.0.0? I thought we had decided it would go into 13.1, with updates for both GCClassic and GCHP at the same time.

LiamBindle · 2020-09-25T19:24:00Z

It's okay if this goes in after 13.0.0. There isn't a dev/13.1.0 branch yet, so I put the target on dev/13.0.0.

A mechanism like init_rd.sh is going to be extremely helpful to my timing tests/CS archive/transport tracers work, so I kind needed something like this anyways in the meantime. I wanted to open the PR so that (1) it's available for my AIST work (via cherry-picking; since I'll be working on 13.0.0 prereleases), and (2) so that further run directory creation updates can build upon this.

Implemented init_rd.sh which enables automated run directory creation. The run directory is initialized based on RDI values loaded from the input files. See #459.

LiamBindle · 2020-09-25T19:52:25Z

I cleaned up the commit log.

Resolved conflicts in: run/GCHP/ExtData.rc.templates/ExtData.rc.fullchem run/GCHP/createRunDir.sh run/GCHP/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

msulprizio · 2021-04-02T19:14:20Z

I just updated this branch to 13.1.0-alpha.2 and resolved conflicts. We still need to incorporate these changes into run/GCClassic for consistency between GCClassic and GCHP.

@lizziel Please review the latest changes before I bring into dev. I've added you as a reviewer.

@liam Do you have any objections to us removing the RDI_ prefix from variables? Both Lizzie and I found it wasn't immediate what that stood for (without looking at the summary above).

The files containing environment variables for various run directory settings will ideally be used for both GCHP and GCClassic. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

…r.sh Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

LiamBindle · 2021-04-02T20:37:07Z

run/GCHP/init_rd.sh

+THIS_SCRIPTS_DIRECTORY=$(realpath $(dirname "$0"))
+
+if [[ ( $* == --rdi-vars ) ]]; then 
+   grep -roh 'RDI_[A-Z_][A-Z_]*' $THIS_SCRIPTS_DIRECTORY | grep -v 'RDI_VARS' |  sort | uniq


This is the line that prints all the run directory initialization variables

LiamBindle · 2021-04-02T20:38:54Z

@msulprizio I think a unique prefix is useful for tracking what templating variables exist in the template directories (and for differentiating them from things like shell variables in runConfig.sh). For example, if you run init_rd.sh --rdi-vars it prints a list of all the template variables, and this depends on that unique prefix.

This way we don't need to keep a list of all the variables that are valid; instead we know it's any variable that start with RDI_. It's a fairly primative approach, but it lets us avoid introducing new dependencies (e.g., templating via python), and it isn't public-facing.

Would an alternative unique prefix be prefered? What do you think?

Resolved conflicts in: run/GCClassic/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem run/GCClassic/HISTORY.rc.templates/HISTORY.rc.fullchem run/GCClassic/createRunDir.sh run/GCClassic/input.geos.templates/input.geos.fullchem run/GCHP/ExtData.rc.templates/ExtData.rc.TransportTracers run/GCHP/ExtData.rc.templates/ExtData.rc.fullchem run/GCHP/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem run/GCHP/createRunDir.sh run/GCHP/input.geos.templates/input.geos.TransportTracers run/GCHP/input.geos.templates/input.geos.fullchem run/GCHP/runConfig.sh.template Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

Made the following changes: - Replaced tokens in configuration files with RDI variables - Removed settings files for several simulation types because they contained RDI variables that are set in createRunDir.sh already - Added settings file for GCAP2 (run/shared/settings/modele2.1.txt) - Updated GCHP adjoint configuration file and files for GCHP CO2 simulation so they are consistent with recent updates Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

…se RDI variables instead Recent modifications for GCAP 2.0 support added several more uses of "sed" in createRunDir.sh. These have been removed where possible and replaced with RDI variables that will be automatically replaced. For HEMCO settings specific to simulations using GMAO or GCAP2 meteorology, the RDI settings have been set in gmao_hemco.txt and gcap2_hemco.txt which are accessed in createRunDir.sh. Similar updates were also made to setupConfigFiles.sh to remove uses of sed from function set_common_settings. That function is still used to manually add species and modify diagnostic output for specific simulation types. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

Run directory variables were previously prefixed with RDI (Run Directory Initialization). It was not immediately obvious what RDI stood for, so we have replaced RDI with the more descriptive RUNDIR. This is a simple swap where all uses of RDI, rdi, or RDI_ have been replaced with RUNDIR, rundir, or RUNDIR_. The run directory variables are now saved to a file named rundir_vars.txt. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

To remove differences in run directory files made with the automated run directory creation updates, the following fixes have been made: 1. Make sure all tokens in HEMCO_Config.rc files are replaced with $RUNDIR_ variables. 2. Remove unused HEMCO settings specific to GCAP 2.0 simulations from the top of HEMCO_Config.rc for several specialty simulations. 3. Make sure to use $GCAP2SCENARIO instead of $SCENARIO for CMIP6 fields consistently throughout HEMCO_Config.rc files. 4. Add dummy HEMCO_Diagn.rc file for tagO3 simulation. 5. In createRunDir.sh set default DustDead tuning factor to -999.0 for resolutions without a recommended scaling. Also make sure to set TOMAS extensions to off when not using that simulation. 6. Fixed a typo for the complexSOA_SVPOA simulation in createRunDir.sh. 7. Make sure to use ${RUNDIR_POP_SPC} instead of {POPs_SPC} in POPs template files. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

…rectory creation updates 1. Fix typo in createRunDir.sh for determining DustDead tuning factor used in 4x5 GEOS-FP benchmark and TOMAS simulations (both of which use online dust emissions). 2. Update commonFunctionsForTests.sh used in Integration Tests to add "NA" to the entries for met fields in the new HEMCO_Config.rc.gmao_metfields file as well as the original HEMCO_Config.rc file. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

…un directory creation updates 1. Fix typos in createRunDir.sh for GCHP. 2. Add RUNDIR_MET_DIR_NATIVE for native-resolution meteorology fields used by GCHP. GCClassic typically uses meteorology fields at the same resolution as the simulation. 3. In commonFunctionsForTests.sh only modify HEMCO_Config.rc.gmao_metfields if the file exists (i.e. only for GCClassic run directories). For GCHP runs, meteorology fields are specified in ExtData.dat and don't need to be modified for nested-grid simulations. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

msulprizio · 2021-10-28T18:40:08Z

These updates are now ready for further review and for merging into the standard code. I made extensive changes since @LiamBindle's initial PR, primarily to expand this capability to GCClassic as well as GCHP and to include more run directory variables for fields that differ based on meteorology (GEOS-FP, MERRA-2, GCAP 2.0), grid resolution, and/or simulation type. I also renamed RDI variables to RUNDIR variables to clear up confusion about the meaning of RDI.

This PR is no longer a zero-diff for all simulation types since it does resolve some issues that existed in the reference version. For example, for marinePOA, acid uptake, and TOMAS simulations, previous simulations used offline emissions by default when online emissions should have been used to properly simulate the emissions for species in those simulations.

The PR name has been updated to reflect these changes.

These updates pass all GCClassic integration tests:

==============================================================================
GEOS-Chem Classic: Execution Test Results

Using 24 OpenMP threads
Number of execution tests: 80
==============================================================================

...

Summary of test results:
------------------------------------------------------------------------------
Execution tests passed: 80
Execution tests failed: 0
Execution tests not yet completed: 0

and GCHP integration tests:

==============================================================================
GCHP: Execution Test Results

Number of execution tests: 3
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gchp_fullchem_benchmark_merra2_c48...............Execute Simulation.....PASS
gchp_fullchem_standard_merra2_c24................Execute Simulation.....PASS
gchp_TransportTracers_geosfp_c24.................Execute Simulation.....PASS
 
Summary of execution test results:
------------------------------------------------------------------------------
Execution tests passed:        3
Execution tests failed:        0
Execution tests not completed: 0

NOTE: There is still room for future work here. Some manipulation of run directory files is still being done in createRunDir.sh, setupConfigFiles.sh, and intTestCreate.sh.

msulprizio · 2021-10-28T18:44:03Z

@LiamBindle I wasn't able to add you as a reviewer since you originally opened this PR, but please do have a look at the changes I've made on top of your original commits and let me know if you have any questions or concerns.

LiamBindle · 2021-10-28T18:45:42Z

@msulprizio Sure thing. I'll review the changes tomorrow.

LiamBindle

@msulprizio Looks good to me. Thanks for spearheading all this!

My only concern is the RUNDIR_VARS that are generated dynamically in createRunDir.sh. What do you think of moving those variables to static .txt files that are included based on the users input?

LiamBindle · 2021-10-29T16:59:18Z

run/GCClassic/createRunDir.sh

@@ -23,6 +23,10 @@ cd ${srcrundir}
 # Load file with utility functions to setup configuration files
 . ${gcdir}/run/shared/setupConfigFiles.sh

+# Initialize run directory variables
+RUNDIR_VARS=""
+RUNDIR_VARS+="RUNDIR_GC_MODE='GCClassic'\n"


Does createRunDir.sh create RUNDIR_VARS that aren't present in static .txt files? If so, is it possible to put these in static .txt files that are then included dynamically?

I just want to clarify because I think it's important that every run directory can be generated from static .txt files, so that there is the option to totally bypass createRunDir.sh.

LiamBindle · 2021-10-29T17:01:00Z

run/GCClassic/createRunDir.sh

+    RUNDIR_VARS+="RUNDIR_USE_ONLINE_O3='F'\n"
+else
+    RUNDIR_VARS+="RUNDIR_USE_NLPBL='T'\n"
+    RUNDIR_VARS+="RUNDIR_USE_ONLINE_O3='T'\n"


Could these be put into static .txt files that are dynamically included based on $sim_extra_option?

LiamBindle · 2021-10-29T17:01:45Z

run/GCClassic/createRunDir.sh

 	elif [[ ${scen_num} = "9" ]]; then
 	    scenario="SSP119"
            runid="E213SSP119aF40oQ40"
-	    met_avail="# 2040-2049; 2090-2099"
+	    RUNDIR_VARS+="RUNDIR_MET_AVAIL='# 2040-2049; 2090-2099'\n"


Same with these

LiamBindle · 2021-10-29T17:02:13Z

run/GCClassic/createRunDir.sh

-	            lon_range="-130.0  -60.0"
-		    lat_range="   9.75  60.0"
+	            RUNDIR_VARS+="RUNDIR_GRID_LON_RANGE='-130.0  -60.0'\n"
+		    RUNDIR_VARS+="RUNDIR_GRID_LAT_RANGE='   9.75  60.0'\n"


Same here, above and below.

LiamBindle · 2021-10-29T17:07:55Z

run/GCHP/createRunDir.sh

+RUNDIR_VARS+="RUNDIR_SIM_EXTRA_OPTION=$sim_extra_option\n"
+
+# Determine settings based on simulation type
+if [[ ${sim_extra_option} == "benchmark"  ]] || \


Similarly to the GC-Classic createRunDir.sh, could we move these to static .txt files that are included based on the user input?

msulprizio · 2021-10-29T22:01:41Z

My only concern is the RUNDIR_VARS that are generated dynamically in createRunDir.sh. What do you think of moving those variables to static .txt files that are included based on the users input?

Yes, I had originally designed it that way, but found there were many overlapping settings between simulation types which led to duplicate variables in the output rundir_vars.txt. I can try to revisit this now that all of the GCClassic simulations have been sorted through and extract these from createRunDirs.sh again. I'll attempt this early next week and keep you posted.

msulprizio · 2021-11-15T16:21:04Z

My only concern is the RUNDIR_VARS that are generated dynamically in createRunDir.sh. What do you think of moving those variables to static .txt files that are included based on the users input?

Yes, I had originally designed it that way, but found there were many overlapping settings between simulation types which led to duplicate variables in the output rundir_vars.txt. I can try to revisit this now that all of the GCClassic simulations have been sorted through and extract these from createRunDirs.sh again. I'll attempt this early next week and keep you posted.

I was not able to address this point before going on maternity leave. I suggest merging as-is in 13.4.0 to avoid further delay on this PR and incurring additional conflicts. We can create a new feature request for this point. As noted above there is still room for other future work here. Some manipulation of run directory files is still being done in createRunDir.sh, setupConfigFiles.sh, and intTestCreate.sh, and this should ideally be incorporated in init_rundir.sh or the .txt files.

yantosca

This is a straightforward PR, it just involves text replacement in the run-directory scripts and configuration files. OK to merge.

One minor update, instead of using rm -f to free symbolic links, I would use unlink, which is perhaps safer. I can make that change.

yantosca · 2021-11-29T21:12:29Z

The feature/automated_rundir_creation branch has been merged up to dev and stored in a new branch dev+automated_rundir_creation. GCHP and GCC integration tests have been scheduled.

yantosca · 2021-11-30T17:14:19Z

I have merged the feature/automated_rundir_creation branch up to dev and stored it in a new branch (dev+automatic_rundir_creation) for testing. The GCC integration tests all passed:

fSummary of test results:
------------------------------------------------------------------------------
Execution tests passed: 80
Execution tests failed: 0
Execution tests not yet completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

but the GCHP tests all failed. The gchp.log file was not created in each of the run directories.

==============================================================================
GCHP: Execution Test Results

Number of execution tests: 3
==============================================================================
 
Execution tests:
------------------------------------------------------------------------------
gchp_fullchem_benchmark_merra2_c48...............Execute Simulation.....FAIL
gchp_fullchem_standard_merra2_c24................Execute Simulation.....FAIL
gchp_TransportTracers_geosfp_c24.................Execute Simulation.....FAIL
 
Summary of execution test results:
------------------------------------------------------------------------------
Execution tests passed:        0
Execution tests failed:        3
Execution tests not completed: 0

Still investigating ...

yantosca · 2021-11-30T17:17:02Z

Here is the rundir_vars.txt from the gchp_TransportTracers_merra2.c24` folder

RUNDIR_ACID_UPTAKE='F'
RUNDIR_AERO_BCPI='* AERO_BCPI      $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_BCPI  2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_BCPO='* AERO_BCPO      $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_BCPO  2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_DST1='* AERO_DST1      $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_DST1  2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_NH4='* AERO_NH4       $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_NH4   2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_NIT='* AERO_NIT       $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_NIT   2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_OCPI='* AERO_OCPI      $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_OCPI  2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_OCPO='* AERO_OCPO      $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_OCPO  2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_AERO_SO4='* AERO_SO4       $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_SO4   2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_Br_GC='* Br_GC          $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_Br    2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_BrO_GC='* BrO_GC         $ROOT/GCClassic_Output/13.0.0/$YYYY/GEOSChem.SpeciesConc.$YYYY$MM$DD_0000z.nc4      SpeciesConc_BrO   2010-2019/1-12/1/0 C xyz 1        * - 1 1'
RUNDIR_BrO_TOMCAT='* BrO_TOMCAT     $ROOT/MERCURY/v2014-09/BrOx/BrOx.GMI.geos5.2x25.nc                                  LBRO2H                 1985/1-12/1/0 C xyz pptv     * - 1 1'
RUNDIR_Br_TOMCAT='* Br_TOMCAT      $ROOT/MERCURY/v2014-09/BrOx/BrOx.GMI.geos5.2x25.nc                                  LBRO2N                 1985/1-12/1/0 C xyz pptv     * - 1 1'
RUNDIR_Bry_DIR='STRAT/v2015-01/Bry'
RUNDIR_CH4_LOSS='* CH4_LOSS       $ROOT/CH4/v2014-09/4x5/gmi.ch4loss.geos5_47L.4x5.nc                                 CH4loss           1985/1-12/1/0      C xyz s-1      * - 1 1'
RUNDIR_CMIP6_FIELDS='false'
RUNDIR_CO2_COPROD='* CO2_COPROD     $ROOT/CO2/v2019-02/CHEM/CO2_prod_rates.GEOS5.2x25.47L.nc                            LCO               2004-2009/1-12/1/0 C xyz kgC/m3/s * - 1 1'
RUNDIR_COMPLEX_SOA='F'
RUNDIR_CORES_PER_NODE='48'
RUNDIR_CS_RES='48'
RUNDIR_DATA_ROOT=/n/holyscratch01/external_repos/GEOS-CHEM/gcgrid/data/ExtData
RUNDIR_DUSTALK_EXT='off'
RUNDIR_DUSTDEAD_EXT='off'
RUNDIR_GCAP2_VERTRES='47'
RUNDIR_GC_MODE='GCHP'

yantosca · 2021-11-30T21:09:53Z

I applied the fix in commit 43f90e7, which fixed the issue. The GCHP integration tests were failing because symbolic links to the various data directories had been removed inadvertently. This is now fixed and the GCHP integration tests all pass:

Summary of execution test results:
------------------------------------------------------------------------------
Execution tests passed:        3
Execution tests failed:        0
Execution tests not completed: 0

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%  All execution tests passed!  %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Implemented init_rd.sh

c3d47be

Implemented init_rd.sh which enables automated run directory creation. The run directory is initialized based on RDI values loaded from the input files. See #459.

LiamBindle force-pushed the feature/automated-rundir-creation branch from 59e85eb to c3d47be Compare September 25, 2020 19:51

Merge branch 'dev/13.0.0' into feature/automated-rundir-creation

e77ff2b

lizziel added this to the 13.1.0 milestone Oct 7, 2020

lizziel mentioned this pull request Oct 21, 2020

[FEATURE REQUEST] Allow createRunDir.sh to run silently (no interactive questions) geoschem/gchp_legacy#38

Closed

yantosca closed this Dec 11, 2020

yantosca deleted the branch dev December 11, 2020 21:24

msulprizio reopened this Dec 11, 2020

msulprizio changed the base branch from dev/13.0.0 to main December 11, 2020 21:30

yantosca added the never stale Never label this issue as stale label Jan 13, 2021

yantosca changed the title ~~Zero-diff run directory update enabling automated run directory creation for GCHP~~ [PULL REQUEST] Zero-diff run directory update enabling automated run directory creation for GCHP Jan 13, 2021

LiamBindle mentioned this pull request Feb 17, 2021

added listInputDataFiles utility script to GCHP run directory #622

Merged

msulprizio assigned LiamBindle Mar 3, 2021

msulprizio added category: Feature Request New feature or request and removed never stale Never label this issue as stale labels Mar 3, 2021

msulprizio changed the base branch from main to dev March 29, 2021 16:52

msulprizio self-requested a review March 29, 2021 16:52

LiamBindle mentioned this pull request Apr 1, 2021

Is this going to be a run directory option in GCHP? geoschem/MinimumStandaloneTransportTracer#1

Open

msulprizio approved these changes Apr 2, 2021

View reviewed changes

msulprizio requested a review from lizziel April 2, 2021 18:22

Merge branch 'dev' into feature/automated-rundir-creation

38f5294

Resolved conflicts in: run/GCHP/ExtData.rc.templates/ExtData.rc.fullchem run/GCHP/createRunDir.sh run/GCHP/HEMCO_Config.rc.templates/HEMCO_Config.rc.fullchem Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

msulprizio added 2 commits April 2, 2021 15:30

Move files from run/GCHP/settings to run/shared/

6e80dea

The files containing environment variables for various run directory settings will ideally be used for both GCHP and GCClassic. Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

Add bug fixes to restore GCHP run directory creation with createRunDi…

a28639a

…r.sh Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

LiamBindle commented Apr 2, 2021

View reviewed changes

msulprizio added 7 commits October 6, 2021 15:19

Add several fixes for the automated run directory creation updates

4389a25

Signed-off-by: Melissa Sulprizio <mpayer@seas.harvard.edu>

msulprizio mentioned this pull request Oct 27, 2021

[PULL REQUEST] Add fix for species indexing in sea salt extension when using marinePOA option geoschem/HEMCO#115

Merged

msulprizio changed the title ~~[PULL REQUEST] Zero-diff run directory update enabling automated run directory creation for GCHP~~ [PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic Oct 27, 2021

msulprizio added this to the 13.4.0 milestone Oct 28, 2021

msulprizio requested a review from yantosca October 28, 2021 18:41

msulprizio self-assigned this Oct 28, 2021

LiamBindle commented Oct 29, 2021

View reviewed changes

msulprizio mentioned this pull request Nov 15, 2021

[FEATURE REQUEST] Use degassing-only volcano climatology for GEOS-Chem benchmarks #1005

Closed

yantosca approved these changes Nov 29, 2021

View reviewed changes

yantosca added topic: Configuration Files Related to GEOS-Chem configuration files no-diff-to-benchmark This update will not change the results of fullchem benchmark simulations labels Nov 29, 2021

yantosca merged commit 966906b into dev Dec 3, 2021

yantosca deleted the feature/automated-rundir-creation branch December 3, 2021 17:37

msulprizio mentioned this pull request Apr 20, 2022

[BUG/ISSUE] Need to update HEMCO run directory creation script to utilize RUNDIR_VARS geoschem/HEMCO#142

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic #459

[PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic #459

LiamBindle commented Sep 25, 2020 •

edited

LiamBindle commented Sep 25, 2020

lizziel commented Sep 25, 2020

LiamBindle commented Sep 25, 2020 •

edited

LiamBindle commented Sep 25, 2020

msulprizio commented Apr 2, 2021

LiamBindle Apr 2, 2021

LiamBindle commented Apr 2, 2021 •

edited

msulprizio commented Oct 28, 2021

msulprizio commented Oct 28, 2021

LiamBindle commented Oct 28, 2021

LiamBindle left a comment •

edited

LiamBindle Oct 29, 2021

LiamBindle Oct 29, 2021

LiamBindle Oct 29, 2021

LiamBindle Oct 29, 2021

LiamBindle Oct 29, 2021

msulprizio commented Oct 29, 2021

msulprizio commented Nov 15, 2021

yantosca left a comment

yantosca commented Nov 29, 2021

yantosca commented Nov 30, 2021 •

edited

yantosca commented Nov 30, 2021

yantosca commented Nov 30, 2021

[PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic #459

[PULL REQUEST] Run directory updates enabling automated run directory creation for GCHP and GCClassic #459

Conversation

LiamBindle commented Sep 25, 2020 • edited

Additional Notes

Example

Co-benefits

Zero-diff tests

Additional tests

LiamBindle commented Sep 25, 2020

lizziel commented Sep 25, 2020

LiamBindle commented Sep 25, 2020 • edited

LiamBindle commented Sep 25, 2020

msulprizio commented Apr 2, 2021

LiamBindle Apr 2, 2021

Choose a reason for hiding this comment

LiamBindle commented Apr 2, 2021 • edited

msulprizio commented Oct 28, 2021

msulprizio commented Oct 28, 2021

LiamBindle commented Oct 28, 2021

LiamBindle left a comment • edited

Choose a reason for hiding this comment

LiamBindle Oct 29, 2021

Choose a reason for hiding this comment

LiamBindle Oct 29, 2021

Choose a reason for hiding this comment

LiamBindle Oct 29, 2021

Choose a reason for hiding this comment

LiamBindle Oct 29, 2021

Choose a reason for hiding this comment

LiamBindle Oct 29, 2021

Choose a reason for hiding this comment

msulprizio commented Oct 29, 2021

msulprizio commented Nov 15, 2021

yantosca left a comment

Choose a reason for hiding this comment

yantosca commented Nov 29, 2021

yantosca commented Nov 30, 2021 • edited

yantosca commented Nov 30, 2021

yantosca commented Nov 30, 2021

LiamBindle commented Sep 25, 2020 •

edited

LiamBindle commented Sep 25, 2020 •

edited

LiamBindle commented Apr 2, 2021 •

edited

LiamBindle left a comment •

edited

yantosca commented Nov 30, 2021 •

edited