Home

These are my personal notes/helpful hints.

For the official page please go to: https://github.com/NOAA-EMC/global-workflow/wiki

Modules

Hera:

module use -a /contrib/anaconda/modulefiles
module load anaconda/anaconda3-5.3.1
module load rocoto/1.3.3

Orion:

module load python/3.7.5
module load contrib
module load rocoto/1.3.3

WCOSS-DELL:

module load python/3.6.3
module use /usrx/local/dev/emc_rocoto/modulefiles
module load rocoto/1.3.0rc2

Standalone

APP=ATM
PSLOT=atm01
BASEDIR=/path/to/area/to/keep/things/
CONFIGDIR=$BASEDIR/global-workflow/parm/config
IDATE=2013040100
EDATE=2013040100
RES=384
GFS_CYC=1
COMROT=$BASEDIR/$PSLOT/COMROOT
EXPDIR=$BASEDIR/$PSLOT/EXPDIR
ICSDIR=$BASEDIR/$PSLOT/ICDIR/${PSLOT}

Run the setup experiment script:

./setup_expt.py forecast-only --app $APP --pslot $PSLOT --configdir $CONFIGDIR --idate $IDATE --edate $EDATE --res $RES --gfs_cyc $GFS_CYC --comrot $COMROT --expdir $EXPDIR --icsdir $ICSDIR

In $EXPDIR/$PSLOT/config.base change the following:

You must change:

DOIAU=NO 

Some things you might want to change:

ACCOUNT
HPSS_PROJECT
KEEPDATA
RUNDIR -- especially if stmp is filling up on hera
ATARDIR -- if you want longer hpss storage etc 

Now run the set up script:

./setup_workflow_fcstonly.py --expdir $EXPDIR/$PSLOT

Your crontab is then: $EXPDIR/$PSLOT/$PSLOT.crontab

Before starting your cronjob, copy your ICS:

Now, set up your ICs from the coupled model.

On Hera:

ICBASEDIR=/scratch2/NCEPDEV/climate/climpara/S2S/IC 

p8b:

ATMIC=GEFS-NoahMP-aerosols-p8a 

Also define:

YYYYMMDD="$(echo $IDATE | cut -c1-8)"
ZZ="$(echo $IDATE | cut -c9-10)"

Make directory and link ICs:

mkdir -p $COMROT/$PSLOT/gfs.${YYYYMMDD}/${ZZ}/atmos/INPUT
ln -sf $ICBASEDIR/$ATMIC/${IDATE}/gfs/C384/INPUT/*  $COMROT/$PSLOT/gfs.${YYYYMMDD}/${ZZ}/atmos/INPUT/

Now you can start your crontab or use rocoto run to step through the rest of the runs.

Coupled

APP=S2SW
PSLOT=try01
BASEDIR=/path/to/area/to/keep/things/
CONFIGDIR=$BASEDIR/global-workflow/parm/config
IDATE=2013040100
EDATE=2013040100
RES=384
GFS_CYC=1
COMROT=$BASEDIR/$PSLOT/COMROOT
EXPDIR=$BASEDIR/$PSLOT/EXPDIR
ICSDIR=$BASEDIR/$PSLOT/ICDIR/${PSLOT}

Run the setup experiment script, for coupling without aerosols:

./setup_expt.py forecast-only --app $APP --pslot $PSLOT --configdir $CONFIGDIR --idate $IDATE --edate $EDATE --res $RES --gfs_cyc $GFS_CYC --comrot $COMROT --expdir $EXPDIR --icsdir $ICSDIR

for coupling with aerosols:

./setup_expt.py forecast-only --app $APP --aerosols --pslot $PSLOT --configdir $CONFIGDIR --idate $IDATE --edate $EDATE --res $RES --gfs_cyc $GFS_CYC --comrot $COMROT --expdir $EXPDIR --icsdir $ICSDIR 

In $EXPDIR/$PSLOT/config.base change the following:

You must change:

DOIAU=NO 

Some things you might want to change:

ACCOUNT
HPSS_PROJECT
KEEPDATA
RUNDIR -- especially if stmp is filling up on hera
ATARDIR -- if you want longer hpss storage etc 

The number of ocean post jobs is controlled in $EXPDIR/$PSLOT/config.ocnpost by NPOSTGRP which should be changed for longer runs.

Cycled tests

Instructions from Walter:

• Go through the normal checkout/build/link. Make sure you do build_all without the -c flag, otherwise you won't build all the GDAS stuff (you can still use coupled with the link script).
• Modify this as needed for your own paths for experiment setup (4½ cycle test):

./setup_expt.py cycled --app ATM --resdet 384 --resens 192 --comrot $NOSCRUB/global-workflow --expdir $SAVE/global-workflow/expdir --idate 2020083000 --edate 2020083112 --nens 80 --gfs_cyc 4 --start warm --pslot $PSLOT

• Go to comrot, delete the $PSLOT directory that was created and then use this command to copy the warm-start files:

cp -as $SOURCE_DIR $PSLOT

Where SOURCE DIR is (the trailing slash is important):

hera: /scratch2/NCEPDEV/ensemble/noscrub/Walter.Kolczynski/global-workflow/C384_warm_IC/
orion: /work2/noaa/global/wkolczyn/noscrub/global-workflow/C384_warm_IC/
dell: /gpfs/dell2/emc/modeling/noscrub/Walter.Kolczynski/global-workflow/C384_warm_IC/

This should copy the directory structure of the source, but all the files will be symlinks.

[Optional to speed things up] In the experiment directory: a. In config.base, edit FHMAX_GFS to be 24. b. In config.resources, edit wtime_fcst_gfs to be "01:00:00" Run setup_workflow.py as normal

[Optional] Edit the workflow XML a. Change the MAXTRIES to 1, so if there is a failure it doesn't try to do it twice b. Change TASKTHROTTLE to "80" so more EnKF jobs run at once Normally I'm just looking for successful completion, but with changes like these (particularly to Thompson MP), we should get someone from the GDAS team to evaluate.