wave configuration update (NOAA-EMC#81)

* Pointing to v3.1 tag for ufs-s2s-model making readme consistant with user.yaml.default * bug fix, POSTGRB2TBL undefined * Squashing commits to wave2global * Fix to JWAVE_PREP to look back a day for rtofs * Moving standalone fv3 model_config exglobal_fcst block into if/else/fi cplwav model_config block. Reinstating config.wave block in JGLOBAL_FORECAST. Pointing EXECwave to HOMEgfs/exec directory for WW3 util executables (changed link_fv3gfs.sh accordingly). Removing debug options from compile.sh line in build_fv3.sh. * Clean up exwave_post_sbs * clean up unused schema move default value into defaults/ * move all default values into /defaults from /schema It's better if we have one place to manage default values for configurable variables other than two Default values only for reference. Please throughly test to make sure it could reproduce. * model_configure_DATM * merge in the latest revision from IPD work * add stoch variables to config.fcst scripts update: CCPP and IPD works * capital case CDUMP values in config.fv3 * stochy variable namelist update, add hard-coded variables into config.fcst (fcst.yaml) * OCNTIM and ICETIM are included in config.fv3 ice variable istep0 bug fix * stochastic variables update * delete falseful copy after namelist and model_configure parsing add FNMSKH conditional clause, coupled vs non-coupled * update on sandbox platform, for FIX_SCRUB settings * turn off stochastic features in case file * configuration update for CCPP scripts * generalize fix file directory for imp_physics 8 * reiflag default value changed to 1 * port to Orion * remove HERA-like paths in platform.yaml of Orion * checkout and External.cfg update * update checkout and External.cfg * case file adjustment for Orion IC * expand gitignore * automate fix file linking, to this location: /work/noaa/marine/jmeixner/tempFixICdir/fix/fix_prep_benchmark3 * updates to checkout, build and link to run prototype 4 * update UPP version to v1.0.8 * making Externals_coupled.cfg consistent with checkout.sh * removing fix from .gitignore * fixed error when linking on hera * replace MOM6IC with BASE_CPLIC for base directory location of IC for s2s model and made it platform dependent. (Orion location is temporary and needs to be replaced, can also be changed in cases/coupled_free_forecast.yaml to users prefered location) * updates for copying ICs * more updates for ICs * updates for new fix (which includes prep for multiple resolutions) and some clean-up * renaming exe to reflect updated build * update diag_table for ocean * updates for ice_in to match prototype 4 * updates for CCPP suite used in p4 * making a fix dir of all available ccpp suite files copying suite xml file to rundirectory * updating for reorganized tar files, pulling in changes from wcoss-p2 workflow * last of archive updates for p4 * updating module files to match s2s model and ncep post modules * we don't want to cycle/have 4 runs per day so setting the cases to be just 1 IC * updating module files on hera * updates to readme for orion * mediator cold start succeeds on hera and forecast job starts post jobs for atmosphere are no longer triggered on cold starts * adding ncks, updating ocean post, some fixes for tar files * updates for tar files and do not keep data * fix typo * updating module_base.orion that allows cold start to run. Not sure if other jobs will now fail though or if what is in your .bashrc file will effect the success/failure of the run. * both forecast and atm post seem to be running okay on orion * wave configuration files * wave configuration update * fix module-setup.sh.inc script, which causes module purge unsuccessful * 1, Moved some of the variables into configuration system from JWAVE scripts 2, Wave configuration system built out 3, Modularized scripts updated to accomodate wave variables * Fix suite name of CCPP Move cplwav and cplwav2atm variable * Revise CCPP_SUITE variable name, default option, to "IPD" * wave post-process job update * reg2grb2 for test on feature coupled-crow * orion hera build test * reg2grb2 Build on Hera and Orion * Merged and ready for test * add ocn-ice build to readme * adding updates to wave scripts from gfsv16b branch by @ajhenrique * cleaning up double cplwav * changes to wave case * bugfix in layout file * bug fix in layout file * move OCNPETS, ICEPETS and WAVEPETS into dedicated model-specific sessions instead of fv3_gfs_settings * first try for adding wave ICs * fix bugs in config.wave and coupled_ic.sh * updates for wave defaults for s2s p4 * updates for running wave jobs * point WW3-related executables through HOMEgfs/fv3_coupled.fd/exec * ocean post hours at boundary are processed twice, now fixed. * wave init job now works on hera * adding missing file from last commit * add ocean, ice and wave configurable variables * add ocean, ice and wave configurable variables * put ic type variable into config.fv3ic * wave init now works on orion * updates to run wave prep for s2s case * updating forecast job for wave jobs * updates for forecast with waves * at this point the ocean post is now running * fix nems.configure for cold start * updated wave-realted scripts to match gfsv16 branch updated environment files env/ORION.env updated configuration system * S2s prototype4 (#2) * add waves to jglobal forecast config * updates - everything but wave post seems to run now, part of wave post does updating model to latest tag * change for lmod -> lmod/lmod on orion updated README * fixing syntax error in JWAVE_PREP job for variables that are not actively being used * adding cycle definition to wave post job * cleanup from duplications Co-authored-by: Jessica.Meixner <Jessica.Meixner@noaa.gov> Co-authored-by: Henrique Alves <henrique.alves@noaa.gov> Co-authored-by: kate.friedman <kate.friedman@noaa.gov> Co-authored-by: Jose-Henrique Alves <47567389+ajhenrique@users.noreply.github.com> Co-authored-by: Lin.Gan <lin.gan@noaa.gov>
JessicaMeixner-NOAA · Jul 20, 2020 · a329e10 · a329e10
1 parent bf34181
commit a329e10
Show file tree

Hide file tree

Showing 71 changed files with 3,671 additions and 458 deletions.
diff --git a/.gitignore b/.gitignore
@@ -165,3 +165,5 @@ util/sub_wcoss_d
 workflow/user.yaml
 sorc/checkout-gldas.fd.log
 sorc/gldas.fd/
+sorc/reg2grb2.fd/*.o
+sorc/reg2grb2.fd/*.mod
diff --git a/README.md b/README.md
@@ -11,8 +11,9 @@ sh checkout.sh coupled                                    # Check out the couple
 ```
 ## Compile code used in ufs-s2s-model and EMC_post and link fixed files and executable programs:
 ```
-sh build_ncep_post.sh        #This command will build ncep_post.
+sh build_ncep_post.sh        #This command will build ncep_post
 sh build_fv3_coupled.sh      #This command will build ufs-s2s-model
+sh build_reg2grb2.sh         #This command will build exes for ocean-ice post
 
 To link fixed files and executable programs for the coupled application:
 On Hera: 
@@ -85,6 +86,13 @@ module load rocoto
 rocotorun -w workflow.xml -d workflow.db
 ```
 
+## Using crons on Orion 
+
+On orion, you will need to update the path to rocotorun in the autogenerated workflow.crontab  to be
+/apps/contrib/rocoto/1.3.1/bin/rocotorun
+
+Also note, on orion you need to be on login node 1 for cronjobs. 
+
 ## Resource handling
 There are two ways of changing resource settings (cpu count, time limits, threads) for a job that has already been defined in the workflow.
 

diff --git a/env/HERA.env b/env/HERA.env
@@ -154,6 +154,11 @@ elif [ $step = "fv3ic" ]; then
     [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
     export APRUN_CHGRES="time"
 
+elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" ]; then
+
+    export mpmd="--multi-prog"
+    export CFP_MP="YES"
+
 elif [ $step = "postsnd" ]; then
 
     nth_max=$(($npe_node_max / $npe_node_postsnd))

diff --git a/env/ORION.env b/env/ORION.env
@@ -189,6 +189,11 @@ elif [ $step = "fv3ic" ]; then
     [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
     export APRUN_CHGRES="time"
 
+elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" ]; then
+
+    export mpmd="--multi-prog"
+    export CFP_MP="YES"
+
 elif [ $step = "postsnd" ]; then
 
     nth_max=$(($npe_node_max / $npe_node_postsnd))

diff --git a/jobs/JGLOBAL_FORECAST b/jobs/JGLOBAL_FORECAST
@@ -9,7 +9,7 @@ date
 #############################
 # Source relevant config files
 #############################
-configs="base fcst"
+configs="base fcst wave"
 export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
 config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
 for config in $configs; do
@@ -62,6 +62,8 @@ if [ $RUN_ENVIR = "nco" ]; then
     export RSTDIR=${GESROOT:?}/$envir
 fi
 
+export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
+export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}
 
 ##############################################
 # Begin JOB SPECIFIC work

diff --git a/jobs/JGLOBAL_FORECAST_MEDCOLD b/jobs/JGLOBAL_FORECAST_MEDCOLD
@@ -9,7 +9,7 @@ date
 #############################
 # Source relevant config files
 #############################
-configs="base fcst"
+configs="base fcst wave"
 export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
 config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
 for config in $configs; do
@@ -62,6 +62,8 @@ if [ $RUN_ENVIR = "nco" ]; then
     export RSTDIR=${GESROOT:?}/$envir
 fi
 
+export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
+export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}
 
 ##############################################
 # Begin JOB SPECIFIC work

diff --git a/jobs/JWAVE_INIT b/jobs/JWAVE_INIT
@@ -7,7 +7,7 @@ set -x -e
 #############################
 # Source relevant config files
 #############################
-configs="base wave waveinit"
+configs="base wave"
 export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
 config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
 for config in $configs; do
@@ -26,18 +26,13 @@ status=$?
 # PATH for working directory
 export NET=${NET:-gfs}
 export RUN=${RUN:-gfs}
-export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
 
-export HOMEgfs=${HOMEgfs:-$NWROOT/${NET}.${gfs_ver}}
+export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
 
 # Add default errchk = err_chk
 export errchk=${errchk:-err_chk}
 
-# Set HOMEwave to HOMEgfs
-HOMEwave=${HOMEwave:-${HOMEgfs}} 
-
-# Create and go to DATA directory
-export DATA=${DATA:-${DATAROOT:?}/${jobid}}
+export DATA=${DATA:-${DATAROOT}/${jobid:?}}
 mkdir -p $DATA
 cd $DATA
 
@@ -50,21 +45,10 @@ sh ./PDY
 
 export pgmout=OUTPUT.$$
 
-export MP_PULSE=0
-
-# Set resources to propagate NTASKS across cfp call
-NTASKS=${NTASKS:-${npe_node_waveinit}}
-export NTASKS=${NTASKS:?NTASKS required to be set}
-
-# Path to HOME Directory
-export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
-export EXECwave=${EXECwave:-$HOMEwave/exec}
-export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
-export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
-export USHwave=${USHwave:-$HOMEwave/ush}
-export EXECcode=${EXECcode:-$CODEwave/exec}
+export COMINice=${COMINice:-$COMROOTp2/omb/prod}
+export COMINwnd=${COMINwnd:-$COMROOT/gfs/prod}
+export COMIN_WAV_CUR=${COMIN_WAV_CUR:-$COMROOTp2/rtofs/prod} 
 
-# Set COM Paths and GETGES environment
 export COMINwave=${COMINwave:-${ROTDIR:?}}
 export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
 export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
@@ -75,19 +59,14 @@ if [ $SENDCOM = YES ]; then
   mkdir -p $COMOUT/rundata
 fi
 
-export wavelog=${COMOUTwave}/wave.log
-
 # Set mpi serial command
-export wavempexec=${wavempexec:-"mpirun -n"}
-export wave_mpmd=${wave_mpmd:-"cfp"}
+export wavempexec=${launcher:-"mpirun -n"}
+export wave_mpmd=${mpmd:-"cfp"}
 
 # Execute the Script  
 $HOMEwave/scripts/exwave_init.sh
 
 # Remove temp directories
-if [ "$KEEPDATA" != "YES" ]; then
-  cd $DATAROOT
-  rm -rf $DATA
-fi
-date
+[[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
+date
diff --git a/jobs/JWAVE_POST_SBS b/jobs/JWAVE_POST_SBS
@@ -7,7 +7,7 @@ set -x -e
 #############################
 # Source relevant config files
 #############################
-configs="base wave wavepostsbs"
+configs="base wave"
 export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
 config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
 for config in $configs; do
@@ -26,72 +26,45 @@ status=$?
 # PATH for working directory
 export NET=${NET:-gfs}
 export RUN=${RUN:-gfs}
-export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
 
-export HOMEgefs=${HOMEgefs:-$NWROOT/$NET.${gefs_ver}}
-export HOMEgfs=${HOMEgfs:-$NWROOT/$NET.${gfs_ver}}
+export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
 
 # Add default errchk = err_chk
 export errchk=${errchk:-err_chk}
 
-# Set HOMEwave to HOMEgefs
-HOMEwave=${HOMEwave:-${HOMEgfs}} 
-
-# Set resources to propagate NTASKS across cfp call
-NTASKS=${NTASKS:-${npe_node_waveprep}}
-export NTASKS=${NTASKS:?NTASKS required to be set}
-
-# Create and go to DATA directory
-export DATA=${DATA:-${DATAROOT:?}/${jobid}}
+export DATA=${DATA:-${DATAROOT}/${jobid:?}}
 mkdir -p $DATA
 cd $DATA
 
-export cyc=${cyc:-00}
+cyc=${cyc:-00}
 export cycle=${cycle:-t${cyc}z}
- 
+
 # Set PDY
 setpdy.sh
 sh ./PDY
 
 export pgmout=OUTPUT.$$
 
-export MP_PULSE=0
+export COMINice=$((COMINice:-$COMROOTp2/omb/prod))
+export COMINwnd=$((COMINwnd:-$COMROOT/gfs/prod))
+export COMIN_WAV_CUR=$((COMIN_WAV_CUR:-$COMROOTp2/rtofs/prod))
 
-# Path to HOME Directory
-export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
-export EXECwave=${EXECwave:-$HOMEwave/exec}
-export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
-export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
-export USHwave=${USHwave:-$HOMEwave/ush}
-export EXECcode=${EXECcode:-$CODEwave/exec}
-
-# Set COM Paths and GETGES environment
 export COMINwave=${COMINwave:-${ROTDIR:?}}
 export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
 export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
 export COMOUT=${COMOUT:-${COMOUTwave}/${COMPONENTwave}.${PDY}/${cyc}}
 
-export COMINice=${COMINice:-${COMROOTp2}/omb/prod}
-export COMINwnd=${COMINwnd:-${COMROOT}/gfs/prod}
-export COMIN_WAV_CUR=${COMIN_WAV_CUR:-${COMROOTp2}/rtofs/prod}
-
 mkdir -p $COMOUT/gridded
 mkdir -p $COMOUT/station
 mkdir -p $COMOUT/stats
 
-export wavelog=${COMOUTwave}/wave.log
-
-# Set mpi serial command
-export wavempexec=${wavempexec:-"mpirun -n"}
-export wave_mpmd=${wave_mpmd:-"cfp"}
-
 env | sort
 
 # Set wave model ID tag to include member number 
 # if ensemble; waveMEMB var empty in deterministic
 # Set wave model ID tag to include member number
 # if ensemble; waveMEMB var empty in deterministic
-membTAG='p'
+export membTAG='p'
 if [ "${waveMEMB}" == "00" ]; then membTAG='c'; fi
 export membTAG
 export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}
@@ -114,4 +87,3 @@ if [ "$KEEPDATA" != "YES" ]; then
   rm -rf $DATA
 fi
 date
-
diff --git a/jobs/JWAVE_PREP b/jobs/JWAVE_PREP
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+set -x
+
 date
 export PS4=' $SECONDS + '
 set -x -e
@@ -26,22 +28,14 @@ status=$?
 # PATH for working directory
 export NET=${NET:-gfs}
 export RUN=${RUN:-gfs}
-export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
 
-export HOMEgfs=${HOMEgfs:-$NWROOT/gfs.${gfs_ver}}
+export COMPONENTwave=${COMPONENTwave:-${RUN}wave}
+export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}
 
 # Add default errchk = err_chk
 export errchk=${errchk:-err_chk}
 
-# Set HOMEwave to HOMEgfs
-HOMEwave=${HOMEwave:-${HOMEgfs}} 
-
-# Set resources to propagate NTASKS across cfp call
-NTASKS=${NTASKS:-${npe_node_waveprep}}
-export NTASKS=${NTASKS:?NTASKS required to be set}
-
-# Create and go to DATA directory
-export DATA=${DATA:-${DATAROOT:?}/${jobid}}
+export DATA=${DATA:-${DATAROOT}/${jobid:?}}
 mkdir -p $DATA
 cd $DATA
 
@@ -56,24 +50,10 @@ export RPDY=$PDY
 
 export pgmout=OUTPUT.$$
 
-export MP_PULSE=0
+export COMINice=${COMINice:-${COMROOTp2}/omb/prod}
+export COMINwnd=${COMINwnd:-${COMROOT}/gfs/prod}
+export COMIN_WAV_CUR=${COMIN_WAV_CUR:-${COMROOTp2}/rtofs/prod}
 
-# CDO required for processing RTOFS currents
-# export CDO=${COMROOTp2}/nwprod/rtofs_glo.v1.2.0/bin/cdo
-export CDO=/gpfs/dell2/emc/verification/noscrub/Todd.Spindler/CDO/bin/cdo
-
-# Path to HOME Directory
-export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
-export EXECwave=${EXECwave:-$HOMEwave/exec}
-export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
-export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
-export USHwave=${USHwave:-$HOMEwave/ush}
-export EXECcode=${EXECcode:-$CODEwave/exec}
-
-###################################
-# Set COM Paths and GETGES environment
-###################################
-# Set COM Paths and GETGES environment
 export COMINwave=${COMINwave:-${ROTDIR:?}}
 export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
 export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
@@ -88,7 +68,7 @@ if [ $RUN_ENVIR = "nco" ]; then
     export RPDY=`$NDATE -24 ${PDY}00 | cut -c1-8`
     export COMIN_WAV_CUR=$(compath.py ${WAVECUR_DID}/prod)/${WAVECUR_DID}.${RPDY}
   fi
-else 
+else
   if [ ! -d $DMPDIR/${WAVECUR_DID}.${RPDY} ]; then export RPDY=`$NDATE -24 ${PDY}00 | cut -c1-8`; fi
   if [ ! -L $ROTDIR/${WAVECUR_DID}.${RPDY} ]; then # Check if symlink already exists in ROTDIR
     $NLN $DMPDIR/${WAVECUR_DID}.${RPDY} $ROTDIR/${WAVECUR_DID}.${RPDY}
@@ -104,15 +84,9 @@ if [ $SENDCOM = YES ]; then
   mkdir -p $COMOUT
 fi
 
-export wavelog=${COMOUTwave}/wave.log
-
 # Set mpi serial command
-export wavempexec=${wavempexec:-"mpirun -n"}
-export wave_mpmd=${wave_mpmd:-"cfp"}
-
-# Set wave model ID tag to include member number 
-# if ensemble; waveMEMB var empty in deterministic
-export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}
+export wavempexec=${launcher:-"mpirun -n"}
+export wave_mpmd=${mpmd:-"cfp"}
 
 # Execute the Script  
 $HOMEwave/scripts/exwave_prep.sh
@@ -123,4 +97,4 @@ if [ "$KEEPDATA" != "YES" ]; then
   rm -rf $DATA
 fi
 date
-
+exit 0