Skip to content

Commit

Permalink
Fix directory overwrite bug in NCO mode. (NOAA-EMC#222)
Browse files Browse the repository at this point in the history
Bug fix to get the WE2E tests to work properly in NCO mode.

Summary of modifications:
------------------------
* Modify scripts and WE2E configuration files that run in NCO mode to prevent simultaneously running experiments in NCO mode from writing to the same set of directories.
* Add a WE2E test on the GSD_RAP13km grid.

Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Fix/add/delete comments and/or informational and/or error messages.
(B) Remove unused code and/or fix code indentation.
(C) Set the variables RUN and envir to EXPT_SUBDIR, which when the WE2E test is run will contain the name of the test.  Note that:
  * We can't get the name of the WE2E test from the file name because before running the experiment generation script for this WE2E test, the script that runs the WE2E tests copies this configuration file into ush and renames it config.sh (because that's the name the experiment generation script expects the configuration file to have).  Thus, at the time the experiment generation script is called, the file name no longer contains the name of the test, so it cannot be used to obtain the test name.
  * Even though EXPT_SUBDIR is set to a null string near the beginning of this file, the script that runs the WE2E tests inserts the name of the test into the definition of EXPT_SUBDIR (i.e. it changes the line 'EXPT_SUBDIR=""' into 'EXPT_SUBDIR="test_name"') before calling the experiment generation script.  Thus, EXPT_SUBDIR will be properly set to the name of the test when the experiment generation script is called.
(D) Bug fix - change variable name from LBC_UPDATE_INTVL_HRS to LBC_SPEC_INTVL_HRS.

File-by-file description of modifications:
-----------------------------------------

jobs/JREGIONAL_RUN_POST:
* In the definition of COMOUT, use the new workflow variable COMOUT_BASEDIR that specifies the base directory under which each cycle's UPP output directories will be placed.

scripts/exregional_get_extrn_files.sh:
* Remove this file.  This has been replaced with scripts/exregional_get_extrn_mdl_files.sh.

tests/baseline_configs/config.GSD_RAP13km.sh:
* This is a new WE2E test on the GSD_RAP13km grid using the FV3_GSD_v0 physics suite.

tests/baseline_configs/config.nco_conus.sh: (A), (C)

tests/baseline_configs/config.nco_conus_c96.sh: (A), (C)

tests/baseline_configs/config.regional_001.sh: (A)

tests/baseline_configs/config.regional_002.sh: (A)

tests/baseline_configs/config.regional_003.sh: (A)

tests/baseline_configs/config.regional_004.sh: (A)

tests/baseline_configs/config.regional_005.sh: (A)

tests/baseline_configs/config.regional_006.sh: (A)

tests/baseline_configs/config.regional_007.sh: (A)

tests/baseline_configs/config.regional_008.sh: (A)

tests/baseline_configs/config.regional_009.sh: (A), (B), (C)
* Change the time interval for specifying external model LBCs from 6 hrs to 3 hrs (to match what is done in all the other WE2E tests that use FV3GFS as the external model for LBCs).

tests/baseline_configs/config.regional_010.sh: (A)

tests/baseline_configs/config.regional_011.sh: (A)

tests/baseline_configs/config.regional_012.sh: (A)

tests/baseline_configs/config.regional_013.sh: (A)

tests/baseline_configs/config.regional_014.sh: (A)

tests/baseline_configs/config.regional_015.sh: (A)

tests/baselines_list.txt:
* Add the new GSD_RAP13km test to the full list of WE2E tests.

tests/run_experiments.sh: (A), (B)

ush/config.community.sh: (D)

ush/config.nco.sh: (D)

ush/config_defaults.sh: (A)
* Increase wallclock times for certain tasks since on hera some of the WE2E tests aren't able to complete within the allowed time.

ush/set_predef_grid_params.sh:
* Change the time step on the GSD_RAP13km grid from 90 sec to 50 sec to get a stable forecast.

ush/setup.sh: (A)
* For simplicity, combine two blocks of code that create different directories into a single block (that creates both sets of directories).
* In NCO mode, change defintion of CYCLE_BASEDIR from "$STMP/tmpnwprd/${EMC_GRID_NAME}" to "$STMP/tmpnwprd/$RUN".  This is to make it easier to associate this directory with an experiment (wich can be done by setting RUN to the experiment name in the experiment's configuration file).  That way, two experiments running simultaneously (but having different names) will not end up writing to the same CYCLE_BASEDIR (which currently can happen if the two experiments are using the same grid, i.e. the same value of EMC_GRID_NAME).
* Add COMOUT_BASEDIR as a new secondary workflow variable.  In NCO mode, set this to the base directory under which the output from the RUN_POST_TN task will be placed (i.e. the cycle-independent portion of the RUN_POST_TN task's output directory) and then check whether this directory exists (and deal with it as specified by the variable PREEXISTING_DIR_METHOD).  The goal here is to be able to uniquely associate this directory to an experiment so that simultaneously running experiments do not write their UPP outputs into the same directory (and possibly overwrite each other's results).  This directory is given by COMOUT_BASEDIR="$COMROOT/$NET/$envir", so we can uniquely associate it with an experiment by setting envir to the name of the experiment (which can be done in the experiment's configuration file).  This is why in the WE2E configuration files listed above (config.nco_conus.sh, config.nco_conus_c96.sh, and config.regional_009.sh), we set envir to the name of the experiment.  In community mode, the new variable COMOUT_BASEDIR is not used, so in that case we set it to an empty string.
  • Loading branch information
gsketefian committed Jun 5, 2020
1 parent b38917b commit fb35daf
Show file tree
Hide file tree
Showing 27 changed files with 437 additions and 1,042 deletions.
2 changes: 1 addition & 1 deletion jobs/JREGIONAL_RUN_POST
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ on the output files corresponding to a specified forecast hour.
#-----------------------------------------------------------------------
#
if [ "${RUN_ENVIR}" = "nco" ]; then
COMOUT="$COMROOT/$NET/$envir/$RUN.$PDY/$cyc"
COMOUT="${COMOUT_BASEDIR}/$RUN.$PDY/$cyc"
postprd_dir="$COMOUT"
else
postprd_dir="${CYCLE_DIR}/postprd"
Expand Down
Loading

0 comments on commit fb35daf

Please sign in to comment.