Restructure ROTDIR for $COMPONENT subfolders

[KateFriedman-NOAA]

To facilitate a structure more conducive to UFS and merging modeling systems together (e.g. GFS + GEFS) the $CYC subfolder of COMROT/ROTDIR needs to be broken into a new $COMPONENT subfolder. This will require all components of the FV3GFS to update.

From discussions with NCO (Steven Earle):

COMOUTatmos, wave, etc = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENTatmos, $COMPONENTwave, etc

obs and all the components prdgen/post will go to:
COMOUT = $COMROOT/$NET/$envir/$RUN.$PDY/$cyc/$COMPONENT where COMPONENT is one of obs, wave, atmos, ocean, etc.

For GFS, NET = gfs, while RUN is either gfs, gdasenkf or gdas

Will break up $CDUMP.$PDY/$CYC into atmos, wave, etc:

• $CDUMP.$PDY/$CYC/atmos
• $CDUMP.$PDY/$CYC/wave

$CDUMP.$PDY/$CYC/wave <--move gdaswave.$PDY to this

Created branch for this work: feature/gfsv16b-restructure
This branch will be merged back into feature/gfsv16b when ready. Will document that merge in issue #1

[KateFriedman-NOAA]

@RussTreadon-NOAA @yangfanglin @JessicaMeixner-NOAA I have committed my initial set of changes to add $COMPONENTatmos and $COMPONENTwave variables to the workflow scripts. My local clone on Mars includes modifications for the components:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

Please feel free to review my initial changes and suggest modifications. I am running a cycled experiment with these changes and will commit further changes as needed. Will merge in recent feature/gfsv16b changes this week as well.

[KateFriedman-NOAA]

First half cycle run, got prep jobs at start of first full cycle to run yesterday, and am awaiting return of Mars (parallel production test) to continue test.

[KateFriedman-NOAA]

(below is fixed based on comment from Fanglin after this was first posted)

Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT:

1. JJOB script for job with multiple components (e.g. forecast):

export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos}
export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}

2. JJOB script for job with single component (e.g. non-forecast jobs):

for atmospheric components:

export COMPONENT=${COMPONENT:-atmos}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

for wave components:

export COMPONENT=${COMPONENT:-wave}
export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}
export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT}

[yangfanglin]

Kate, I thought we should not use export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave} Instead using atmos and wave directly where they are referred. export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave} Fanglin

…

On Thu, Jul 16, 2020 at 12:32 PM Kate Friedman ***@***.***> wrote: Final email with Steven Earle lays out the new $COMPONENT variables and COMIN/COMOUT: 1. JJOB script for job with multiple components (e.g. forecast): export COMPONENTatmos=${COMPONENTatmos:-atmos} export COMPONENTwave=${COMPONENTwave:-wave} export COMINatmos=${COMINatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMOUTatmos=${COMOUTatmos:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTatmos} export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave} export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/$COMPONENTwave} 1. JJOB script for job with single component (e.g. non-forecast jobs): for atmospheric components: export COMPONENT=${COMPONENT:-atmos} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} for wave components: export COMPONENT=${COMPONENT:-wave} export COMIN=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} export COMOUT=${ROTDIR}/${RUN}.${PDY}/${cyc}/${COMPONENT} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKY5N2NQ3JYXI5LLG4NFC7DR34TQXANCNFSM4OLNIMHQ> .

-- Fanglin Yang, Ph.D. Physical Scientist Environmental Modeling Center National Centers for Environmental Prediction 301-6833722; fanglin.yang@noaa.gov http://www.emc.ncep.noaa.gov/gmb/wx24fy/fyang/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

[KateFriedman-NOAA]

Fanglin, you're right, let me adjust that, sorry, got my notes on that special case mixed up.

[KateFriedman-NOAA]

@yangfanglin fixed at 3d9f0dc

[KateFriedman-NOAA]

Summary of changes for FV3/ufs-weather-model component: none, all changes on global-workflow side to both JJOB and ex scripts, no changes needed in ufs-weather-model

Changes to scripts on workflow side can be seen in feature/gfsv16b-restructure branch or here:

JGLOBAL_FORECAST

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/jobs/JGLOBAL_FORECAST jobs/JGLOBAL_FORECAST
72a73,80
> # Set wave variables
> if [ ${DO_WAVE:-"NO"} = "YES" ]; then
>   # WAVE component directory
>   export CDUMPwave=${CDUMPwave:-${CDUMP}wave}
>   export COMINwave=${COMINwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
>   export COMOUTwave=${COMOUTwave:-${ROTDIR}/${CDUMP}.${PDY}/${cyc}/wave}
> fi
>

exglobal_fcst_nemsfv3gfs.sh

[Kate.Friedman@m72a3 feature-gfsv16b-restructure]$ diff ../feature-gfsv16b/scripts/exglobal_fcst_nemsfv3gfs.sh scripts/exglobal_fcst_nemsfv3gfs.sh
42a43
> CDUMPwave="${CDUMP}wave"
153,157c154
<     if [ $CDUMP = "gdas" ]; then
<         RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart
<     else
<         RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart}
<     fi
---
>     RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart
163c160
<     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/RERUN_RESTART
---
>     RSTDIR_ATM=${RSTDIR:-$ROTDIR}/${CDUMP}.${PDY}/${cyc}/atmos/RERUN_RESTART
205c202
< memdir=$ROTDIR/${prefix}.$PDY/$cyc/$memchar
---
> memdir=$ROTDIR/${prefix}.$PDY/$cyc/atmos/$memchar
211c208
< gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/$memchar
---
> gmemdir=$ROTDIR/${rprefix}.$gPDY/$gcyc/atmos/$memchar
413d409
< #-------------wavewave----------------------
415c411,412
<   for file in $(ls $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/rmp_src_to_dst_conserv_*) ; do
---
>   # Link WW3 files
>   for file in $(ls $COMINwave/rundata/rmp_src_to_dst_conserv_*) ; do
418c415
<   $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/ww3_multi.${CDUMP}wave${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
---
>   $NLN $COMINwave/rundata/ww3_multi.${CDUMPwave}${WAV_MEMBER}.${cycle}.inp $DATA/ww3_multi.inp
419a417
>   # Check for expected wave grids for this run
421a420
>
423c422,424
<     $NLN $ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.mod_def.$wavGRD $DATA/mod_def.$wavGRD
---
>     # Wave IC (restart) file must exist for warm start on this cycle, if not wave model starts from flat ocean
>     # For IAU needs to use sPDY for adding IAU backup of 3h
>     $NLN $COMINwave/rundata/${CDUMPwave}.mod_def.$wavGRD $DATA/mod_def.$wavGRD
426,432c427,435
<   WAVHCYC=${WAVHCYC:-6}
<   WRDATE=`$NDATE -${WAVHCYC} $CDATE`
<   WRPDY=`echo $WRDATE | cut -c1-8`
<   WRcyc=`echo $WRDATE | cut -c9-10`
<   WRDIR=$ROTDIR/gdaswave.${WRPDY}/${WRcyc}/restart
<   datwave=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/
<   wavprfx=${CDUMP}wave${WAV_MEMBER}
---
>   # Wave IC (restart) interval assumes 4 daily cycles (restarts only written by gdas cycle)
>   # WAVHCYC needs to be consistent with restart write interval in ww3_multi.inp or will FAIL
>   export WAVHCYC=${WAVHCYC:-6}
>   export WRDATE=`$NDATE -${WAVHCYC} $CDATE`
>   export WRPDY=`echo $WRDATE | cut -c1-8`
>   export WRcyc=`echo $WRDATE | cut -c9-10`
>   export WRDIR=${ROTDIR}/${CDUMPRSTwave}.${WRPDY}/${WRcyc}/wave/restart
>   export datwave=$COMOUTwave/rundata
>   export wavprfx=${CDUMPwave}${WAV_MEMBER}
444c447
<     wavicefile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVEICE_FID}.${cycle}.ice
---
>     wavicefile=$COMINwave/rundata/${CDUMPwave}.${WAVEICE_FID}.${cycle}.ice
454c457
<     wavcurfile=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/rundata/${CDUMP}wave.${WAVECUR_FID}.${cycle}.cur
---
>     wavcurfile=$COMINwave/rundata/${CDUMPwave}.${WAVECUR_FID}.${cycle}.cur
465a469,470
>
>   # Loop for gridded output (uses FHINC)
491a497
> fi #cplwav=true
493,495d498
< fi
< #-------------wavewave----------------------
<

[yangfanglin]

Kate,

The structure of the following block needs to be preserved for restarting wave model from a breakpoint. NCO will save breakpoint restart files for the gfs cycle in a $nwges directory , while restart files for the gdas cycle in $comrot. So in operation RSTDIR_WAVE for the gfs cycle will be defined in an ecflow suite definition file. This is also true for the atmos part.

< if [ $CDUMP = "gdas" ]; then
< RSTDIR_WAVE=$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart
< else
< RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}wave.${PDY}/${cyc}/restart}
< fi

You can change this block to

if [ $CDUMP = "gdas" ]; then
RSTDIR_WAVE=$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart
else
RSTDIR_WAVE=${RSTDIR_WAVE:-$ROTDIR/${CDUMP}.${PDY}/${cyc}/wave/restart}
fi

[KateFriedman-NOAA]

Fanglin, gotcha, changed it to the block you provided at 273c535, thanks!

[KateFriedman-NOAA]

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure
Changes committed: https://github.com/KateFriedman-NOAA/GSI/commit/f6bc0f6a010b0606bf6003f14872fb3ddc80f96f

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure
Changes committed: https://github.com/KateFriedman-NOAA/GLDAS/commit/ca6188375dfb46ac8b59346437c1b96ad50ea708

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure
Changes committed: https://github.com/KateFriedman-NOAA/EMC_post/commit/9cb2723b00c276a2e004a64a4bebf1e0e3f943ca

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure
Changes committed: https://github.com/KateFriedman-NOAA/EMC_gfs_wafs/commit/52ed70d8351d186f04e25b5fe87f3bfb38b7fa5e

[KateFriedman-NOAA]

Continuing to let test run through additional cycles. Test on Mars:

EXPDIR: /gpfs/dell2/emc/modeling/save/Kate.Friedman/expdir/testrestrucb
COMROT: /gpfs/dell3/ptmp/Kate.Friedman/comrot/testrestrucb
clone: /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-gfsv16b-restructure

[RussTreadon-NOAA]

Quick look at GFS v16 DA changes looks good. $COMPONENT has been added to DA j-jobs and two exglobal scripts. COMPONENT defaults to "atmos" if not externally set. Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

[KateFriedman-NOAA]

Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel.

I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from:

$CDUMPwave.$PDY/$cyc
...to:
$CDUMP.$PDY/$CYC/wave

[RussTreadon-NOAA]

OK. I'll wait for and process the PR at the appropriate time.

…

On Thu, Jul 16, 2020 at 2:50 PM Kate Friedman ***@***.***> wrote: Seems we could pull these changes into base release/gfsda.v16.0.0 with COMPONENT="" and GFS v16 DA would work as-is in current real-time v16 parallel. I agree and am suggesting that path in the email I'm drafting to everyone. The only part that will need more of a pivot is the wave part since it's moving from: $CDUMPwave.$PDY/$cyc ...to: $CDUMP.$PDY/$CYC/wave — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNN63ZCRRL3KGTE2FMIC3DR35DVFANCNFSM4OLNIMHQ> .

[GeorgeGayno-NOAA]

Summary of changes to other FV3GFS components:

GSI:

Fork branch: release/gfsda.v16.0.0-restructure
Changes committed: KateFriedman-NOAA/GSI@f6bc0f6

GLDAS:

Fork branch: gldas_gfsv16_release.v1.3.0-restructure
Changes committed: KateFriedman-NOAA/GLDAS@ca61883

EMC_post:

Fork branch: upp_gfsv16_release.v1.0.10-restructure
Changes committed: KateFriedman-NOAA/EMC_post@9cb2723

UFS_UTILS:

Just a three line change to ush/fv3gfs_chgres.sh to add $COMPONENT. We're moving away from that script though right? If so, no changes needed to UFS_UTILS.

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

WAFS:

Fork branch: gfs_wafs.v5.0.11-restructure
Changes committed: KateFriedman-NOAA/EMC_gfs_wafs@52ed70d

[KateFriedman-NOAA]

The ./ush/fv3gfs_chgres.sh script is used by the old chgres and is obsolete. It has been replaced by the scripts in ./util/gdas_init.

Thanks for the confirmation that fv3gfs_chgres.sh is indeed obsolete now! I'll forgo submitting changes to you for fv3gfs_chgres.sh then, thanks @GeorgeGayno-NOAA !

[KateFriedman-NOAA]

Global-workflow PR completed: #109