Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/metplus #26

Merged
merged 11 commits into from
Mar 4, 2020
Merged

Feature/metplus #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1915aa9
Issue #8 - pulled in config.metp and modifications to two setup scripts
KateFriedman-NOAA Jan 10, 2020
9c94156
Changes to setup_workflow.py for gfsmetp metatask
malloryprow Jan 21, 2020
6872f79
Add rocoto METplus job script
malloryprow Jan 21, 2020
72e8adf
Update EMC_verif-global tag checkout to 1.4.0
malloryprow Jan 21, 2020
82e6907
Update config.resources for gfsmetp
malloryprow Jan 21, 2020
c0d7179
Add updated env machine files for gfsmetp
malloryprow Jan 21, 2020
0ea809c
Update file format variable in config.metp of online archive files
malloryprow Jan 23, 2020
c11dfef
Update EMC_verif-global tag to verif_global_v1.4.1
malloryprow Jan 24, 2020
786806f
Missed file format updates in a few places in config.metp
malloryprow Jan 24, 2020
eb73e52
Update EMC_verif-global checkout to verif_global_v1.5.0
malloryprow Feb 7, 2020
31bb7d3
Merge branch 'develop' into feature/metplus
malloryprow Feb 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy"
echo "anal fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down
2 changes: 1 addition & 1 deletion env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy"
echo "anal fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down
7 changes: 6 additions & 1 deletion env/WCOSS_C.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy"
echo "anal fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down Expand Up @@ -166,6 +166,11 @@ elif [ $step = "vrfy" ]; then
export IOBUF_PARAMS="*:size=32M:count=4:verbose"
export APRUNTRACK="$launcher -j1 -n1 -N1 -d1 -cc depth"

elif [ $step = "metp" ]; then

export IOBUF_PARAMS="*:size=32M:count=4:verbose"
export APRUNTRACK="$launcher -j1 -n1 -N1 -d1 -cc depth"

Comment on lines +169 to +173
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checking the metp block isn't needed in the other env files. I only see this added to the WCOSS_C env file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't be. It has been working fine in GFSv16, and this env file was the only one with a vrfy block so just to be similar to vrfy I added the metp block.

elif [ $step = "postsnd" ]; then
export IOBUF_PARAMS="sigf*:size=128M:count=20:prefetch=0:verbose,gfs_collectiv*:size=128M:count=2:prefetch=0:verbose,*.snd:size=128M:count=3:prefetch=0:verbose,*.sfc:size=32M:count=3:prefetch=0:verbose,bufr.*:size=8M:count=20:prefetch=0:verbose"

Expand Down
2 changes: 1 addition & 1 deletion env/WCOSS_DELL_P3.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy"
echo "anal fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down
71 changes: 71 additions & 0 deletions jobs/rocoto/metp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/bin/ksh -x

###############################################################
## Abstract:
## Inline METplus verification and diagnostics driver script
## RUN_ENVIR : runtime environment (emc | nco)
## HOMEgfs : /full/path/to/workflow
## EXPDIR : /full/path/to/config/files
## CDATE : current analysis date (YYYYMMDDHH)
## CDUMP : cycle name (gdas / gfs)
## PDY : current date (YYYYMMDD)
## cyc : current cycle (HH)
## METPCASE : METplus verification use case (g2g1 | g2o1 | pcp1)
###############################################################

###############################################################
echo
echo "=============== START TO SOURCE FV3GFS WORKFLOW MODULES ==============="
. $HOMEgfs/ush/load_fv3gfs_modules.sh
status=$?
[[ $status -ne 0 ]] && exit $status


###############################################################
echo
echo "=============== START TO SOURCE RELEVANT CONFIGS ==============="
configs="base vrfy metp"
for config in $configs; do
. $EXPDIR/config.${config}
status=$?
[[ $status -ne 0 ]] && exit $status
done


###############################################################
echo
echo "=============== START TO SOURCE MACHINE RUNTIME ENVIRONMENT ==============="
. $BASE_ENV/${machine}.env metp
status=$?
[[ $status -ne 0 ]] && exit $status

###############################################################
export CDATEm1=$($NDATE -24 $CDATE)
export PDYm1=$(echo $CDATEm1 | cut -c1-8)

export COMIN="$ROTDIR/$CDUMP.$PDY/$cyc"
export DATAROOT="$RUNDIR/$CDATE/$CDUMP/vrfy"
[[ -d $DATAROOT ]] && rm -rf $DATAROOT
mkdir -p $DATAROOT


###############################################################
echo
echo "=============== START TO RUN METPLUS VERIFICATION ==============="
if [ $CDUMP = "gfs" ]; then

if [ $RUN_GRID2GRID_STEP1 = "YES" -o $RUN_GRID2OBS_STEP1 = "YES" -o $RUN_PRECIP_STEP1 = "YES" ]; then

$VERIF_GLOBALSH

fi
fi


if [ $CDUMP = "gdas" ]; then
echo "METplus verification currently not supported for CDUMP=${CDUMP}"
fi
###############################################################
# Force Exit out cleanly
if [ ${KEEPDATA:-"NO"} = "NO" ] ; then rm -rf $DATAROOT ; fi
exit 0
69 changes: 69 additions & 0 deletions parm/config/config.metp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/bin/ksh -x

########## config.metp ##########
# METplus verification step specific

echo "BEGIN: config.metp"

# Get task specific resources
. $EXPDIR/config.resources metp

export RUN_GRID2GRID_STEP1="YES" # Run grid-to-grid verification using METplus
export RUN_GRID2OBS_STEP1="YES" # Run grid-to-obs verification using METplus
export RUN_PRECIP_STEP1="YES" # Run precip verification using METplus


#----------------------------------------------------------
# METplus, Verify grid-to-grid, and/or grid-to-obs, and/or precipitation options
#----------------------------------------------------------

if [ "$CDUMP" = "gfs" ] ; then
if [ $RUN_GRID2GRID_STEP1 = "YES" -o $RUN_GRID2OBS_STEP1 = "YES" -o $RUN_PRECIP_STEP1 = "YES" ]; then
export HOMEverif_global=${HOMEgfs}/sorc/verif-global.fd
export VERIF_GLOBALSH=$HOMEverif_global/ush/run_verif_global_in_global_workflow.sh
## INPUT DATA SETTINGS
export model_list=$PSLOT
export model_data_dir_list=$ARCDIR/..
export model_fileformat_list="pgbf{lead?fmt=%H}.${CDUMP}.{init?fmt=%Y%m%d%H}"
export model_hpssdir_list=$ATARDIR/..
export get_data_from_hpss="NO"
export hpss_walltime="10"
## OUTPUT SETTINGS
export OUTPUTROOT=$RUNDIR/$CDUMP/$CDATE/vrfy/metplus_exp
export model_arch_dir_list=$ARCDIR/..
export make_met_data_by="VALID"
export gather_by="VSDB"
## DATE SETTINGS
export VRFYBACK_HRS="24"
## METPLUS SETTINGS
export METplus_verbosity="INFO"
export MET_verbosity="2"
export log_MET_output_to_METplus="yes"
## FORECAST VERIFICATION SETTINGS
export fhr_min=$FHMIN_GFS
export fhr_max=$FHMAX_GFS
# GRID-TO-GRID STEP 1
export g2g1_type_list="anom pres sfc"
export g2g1_anl_name="self_anl"
export g2g1_anl_fileformat_list="pgbanl.gfs.{valid?fmt=%Y%m%d%H}"
export g2g1_grid="G002"
# GRID-TO-OBS STEP 1
export g2o1_type_list="upper_air conus_sfc"
export g2o1_obtype_upper_air="ADPUPA"
export g2o1_grid_upper_air="G003"
export g2o1_fhr_out_upper_air="6"
export g2o1_obtype_conus_sfc="ONLYSF"
export g2o1_grid_conus_sfc="G104"
export g2o1_fhr_out_conus_sfc="3"
export g2o1_prepbufr_data_runhpss="YES"
# PRECIP STEP 1
export precip1_obtype="ccpa"
export precip1_accum_length="24"
export precip1_model_bucket_list="06"
export precip1_model_varname_list="APCP"
export precip1_model_fileformat_list="pgbf{lead?fmt=%H}.gfs.{init?fmt=%Y%m%d%H}"
export precip1_grid="G211"
fi
fi

echo "END: config.metp"
17 changes: 16 additions & 1 deletion parm/config/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ if [ $# -ne 1 ]; then

echo "Must specify an input task argument to set resource variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy arch"
echo "anal fcst post vrfy metp arch"
echo "eobs eomg eupd ecen efcs epos earc"
echo "postsnd awips gempak"
exit 1
Expand Down Expand Up @@ -112,6 +112,21 @@ elif [ $step = "vrfy" ]; then
export memory_vrfy="16384M"
fi

elif [ $step = "metp" ]; then

export nth_metp=1
export wtime_metp="03:00:00"
export npe_metp=4
export npe_node_metp=4
export wtime_metp_gfs="06:00:00"
export npe_metp_gfs=4
export npe_node_metp_gfs=4
if [[ "$machine" == "WCOSS_C" ]]; then
export memory_metp="3072M"
elif [[ "$machine" == "THEIA" ]]; then
export memory_metp="16384M"
fi

elif [ $step = "arch" -o $step = "earc" -o $step = "getic" ]; then

eval "export wtime_$step='06:00:00'"
Expand Down
109 changes: 24 additions & 85 deletions parm/config/config.vrfy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,20 +14,16 @@ export CDFNL="gdas" # Scores verification against GDAS/GFS analysi

export MKPGB4PRCP="YES" # Make 0.25-deg pgb files in ARCDIR for precip verification
export VRFYFITS="YES" # Fit to observations
export VRFY_PCKG2RUN="VSDB" # "VSDB" = run VSDB verification package only, "METPLUS" = run METplus verification package only, "BOTH = run both verification packages
export VSDB_STEP1="YES" # Populate VSDB database
export VSDB_STEP2="NO"
export VRFYG2OBS="YES" # Grid to observations, see note below if turning ON
export VRFYPRCP="YES" # Precip threat scores
export VRFYRAD="YES" # Radiance data assimilation monitoring
export VRFYOZN="YES" # Ozone data assimilation monitoring
export VRFYMINMON="YES" # GSI minimization monitoring
export VRFYTRAK="YES" # Hurricane track verification
export VRFYGENESIS="YES" # Cyclone genesis verification
export VRFYTRAK="YES" # Hurricane track verification
export VRFYGENESIS="YES" # Cyclone genesis verification
export RUNMOS="NO" # whether to run entire MOS package
export RUN_METPLUS_GRID2GRID_STEP1="YES" # Run grid-to-grid verification using METplus
export RUN_METPLUS_GRID2OBS_STEP1="YES" # Run grid-to-obs verification using METplus
export RUN_METPLUS_PRECIP_STEP1="YES" # Run precip verification using METplus

#-------------------------------------------------
# Fit to Observations
Expand All @@ -36,7 +32,7 @@ export RUN_METPLUS_PRECIP_STEP1="YES" # Run precip verification using METplus
if [ $VRFYFITS = "YES" ]; then

export PRVT=$HOMEgfs/fix/fix_gsi/prepobs_errtable.global
export HYBLEVS=$HOMEgfs/fix/fix_am/global_hyblev.l65.txt
export HYBLEVS=$HOMEgfs/fix/fix_am/global_hyblev.l${LEVS}.txt
export CUE2RUN=$QUEUE

if [ $machine = "WCOSS_C" ]; then
Expand All @@ -62,84 +58,25 @@ if [ "$CDUMP" = "gfs" ] ; then
ddd=`echo $CDATE |cut -c 1-8`
#if [ $ddd -eq 5 -o $ddd -eq 10 ]; then export VSDB_STEP2 = "YES" ;fi

if [ $VRFY_PCKG2RUN = "VSDB" -o $VRFY_PCKG2RUN = "BOTH" ]; then
if [ $VSDB_STEP1 = "YES" -o $VSDB_STEP2 = "YES" -o $VRFYPRCP = "YES" -o $VRFYG2OBS = "YES" ]; then
export BACKDATEVSDB=24 # execute vsdbjob for the previous day
export VBACKUP_PRCP=24 # back up for QPF verification data
export vsdbsave="$NOSCRUB/archive/vsdb_data" # place to save vsdb database
export vsdbhome=$BASE_VERIF # location of global verification scripts
export VSDBSH="$vsdbhome/vsdbjob.sh" # VSDB job script
export vlength=$FHMAX_GFS # verification length
export vhr_rain=$FHMAX_GFS # verification length for precip
export ftyplist="pgbq" # verif. files used for computing QPF ETS scores
export ptyplist="PRATE" # precip types in GRIB: PRATE or APCP
export anltype="gfs" # default=gfs, analysis type (gfs or gdas) for verification
export rain_bucket=6 # prate in pgb files is 6-hr accumulated

export VSDB_START_DATE="$SDATE" # starting date for vsdb maps
export webhost="emcrzdm.ncep.noaa.gov" # webhost(rzdm) computer
export webhostid="$USER" # webhost(rzdm) user name
export SEND2WEB="NO" # whether or not to send maps to webhost
export WEBDIR="/home/people/emc/www/htdocs/gmb/${webhostid}/vsdb/$PSLOT"
export mdlist="gfs $PSLOT " # exps (up to 10) to compare in maps
fi
fi
fi


#----------------------------------------------------------
# METplus, Verify grid-to-grid, and/or grid-to-obs, and/or precipitation options
#----------------------------------------------------------
# All these call $METPLUSSH

if [ "$CDUMP" = "gfs" ] ; then
if [ $RUN_METPLUS_GRID2GRID_STEP1 = "YES" -o $RUN_METPLUS_GRID2OBS_STEP1 = "YES" -o $RUN_METPLUS_PRECIP_STEP1 = "YES" ]; then
if [ $VRFY_PCKG2RUN = "METPLUS" -o $VRFY_PCKG2RUN = "BOTH" ]; then
export HOMEverif_global=${HOMEgfs}/sorc/verif-global.fd
export VERIF_GLOBALSH=$HOMEverif_global/ush/run_verif_global_in_global_workflow.sh
export RUNBATCH="NO"
## INPUT DATA SETTINGS
export model_list=$PSLOT
export model_data_dir_list=$ARCDIR/..
export model_fileformat_list="pgbf{lead?fmt=%H}.${CDUMP}.{init?fmt=%Y%m%d%H}"
export model_hpssdir_list=$ATARDIR/..
export get_data_from_hpss="NO"
export hpss_walltime="10"
## OUTPUT SETTINGS
export OUTPUTROOT=$RUNDIR/$CDUMP/$CDATE/vrfy/metplus_exp
export model_arch_dir_list=$ARCDIR/..
export make_met_data_by="VALID"
export gather_by="VSDB"
## DATE SETTINGS
export VRFYBACK_HRS="24"
## METPLUS SETTINGS
export METplus_verbosity="INFO"
export MET_verbosity="2"
export log_MET_output_to_METplus="yes"
## FORECAST VERIFICATION SETTINGS
export fhr_min=$FHMIN_GFS
export fhr_max=$FHMAX_GFS
# GRID-TO-GRID STEP 1
export g2g1_type_list="anom pres sfc"
export g2g1_anl_name="self_anl"
export g2g1_anl_fileformat_list="pgbanl.gfs.{valid?fmt=%Y%m%d%H}"
export g2g1_grid="G002"
# GRID-TO-OBS STEP 1
export g2o1_type_list="upper_air conus_sfc"
export g2o1_obtype_upper_air="ADPUPA"
export g2o1_grid_upper_air="G003"
export g2o1_fhr_out_upper_air="6"
export g2o1_obtype_conus_sfc="ONLYSF"
export g2o1_grid_conus_sfc="G104"
export g2o1_fhr_out_conus_sfc="3"
# PRECIP STEP 1
export precip1_obtype="ccpa"
export precip1_accum_length="24"
export precip1_model_bucket_list="06"
export precip1_model_varname_list="APCP"
export precip1_model_fileformat_list="pgbf{lead?fmt=%H}.gfs.{init?fmt=%Y%m%d%H}"
export precip1_grid="G211"
fi
if [ $VSDB_STEP1 = "YES" -o $VSDB_STEP2 = "YES" -o $VRFYPRCP = "YES" -o $VRFYG2OBS = "YES" ]; then
export BACKDATEVSDB=24 # execute vsdbjob for the previous day
export VBACKUP_PRCP=24 # back up for QPF verification data
export vsdbsave="$NOSCRUB/archive/vsdb_data" # place to save vsdb database
export vsdbhome=$BASE_VERIF # location of global verification scripts
export VSDBSH="$vsdbhome/vsdbjob.sh" # VSDB job script
export vlength=$FHMAX_GFS # verification length
export vhr_rain=$FHMAX_GFS # verification length for precip
export ftyplist="pgbq" # verif. files used for computing QPF ETS scores
export ptyplist="PRATE" # precip types in GRIB: PRATE or APCP
export anltype="gfs" # default=gfs, analysis type (gfs or gdas) for verification
export rain_bucket=6 # prate in pgb files is 6-hr accumulated

export VSDB_START_DATE="$SDATE" # starting date for vsdb maps
export webhost="emcrzdm.ncep.noaa.gov" # webhost(rzdm) computer
export webhostid="$USER" # webhost(rzdm) user name
export SEND2WEB="NO" # whether or not to send maps to webhost
export WEBDIR="/home/people/emc/www/htdocs/gmb/${webhostid}/vsdb/$PSLOT"
export mdlist="gfs $PSLOT " # exps (up to 10) to compare in maps
fi
fi

Expand Down Expand Up @@ -208,6 +145,8 @@ if [ "$VRFYTRAK" = "YES" ]; then
if [ $machine = "HERA" ]; then
export COMROOTp1="/scratch1/NCEPDEV/rstprod/com"
export COMINsyn=${COMINsyn:-${COMROOTp1}/arch/prod/syndat}
else
export COMINsyn=${COMINsyn:-${COMROOT}/gfs/prod/syndat}
fi
fi

Expand Down
2 changes: 1 addition & 1 deletion sorc/checkout.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ if [[ ! -d verif-global.fd ]] ; then
rm -f ${topdir}/checkout-verif-global.log
git clone --recursive gerrit:EMC_verif-global verif-global.fd >> ${topdir}/checkout-verif-global.log 2>&1
cd verif-global.fd
git checkout verif_global_v1.2.2
git checkout verif_global_v1.5.0
cd ${topdir}
else
echo 'Skip. Directory verif-global.fd already exist.'
Expand Down
Loading