From ea1aa3b5edd65b533dbc4d61cbfa25ff407f6c9a Mon Sep 17 00:00:00 2001 From: "Judy.K.Henderson" Date: Wed, 1 Jul 2020 22:49:56 +0000 Subject: [PATCH] - merge 23Jun20 port2orion branch, ed6b279, into gsd-ccpp-jet - updated INFO file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed commit of the following: commit be717cec4f252ae6949ce4a0a33763de836f3080 Merge: df8dc52e 99277ae3 Author: Kate.Friedman Date: Tue Jun 23 16:20:51 2020 +0000 Issue #5 - Sync merge branch 'feature/gfsv16b' into port2orion * feature/gfsv16b: Update README.md modified: Externals.cfg and sorc/checkout.sh to check out model tag GFS.v16.0.7 and UPP tag upp_gfsv16_release.v1.0.9. Changes include: Update version of upp in Externals.cfg to be consistent with sorc/checkout.sh commit 99277ae34ef50454fa15e7e28b564c2e34e3406c Merge: 14dd3c94 4f8d5a5f Author: fanglin.yang Date: Tue Jun 23 16:05:16 2020 +0000 Merge branch 'feature/gfsv16b' of https://github.com/NOAA-EMC/global-workflow into feature/gfsv16b commit 4f8d5a5f28c7120905fe36e9254f0199da986188 Author: Kate Friedman Date: Tue Jun 23 11:57:26 2020 -0400 Update README.md Remove use/mention of manage_externals until checkout.sh is retired. commit 14dd3c94938b0e69601e25a42104b2fc23944ebd Author: fanglin.yang Date: Tue Jun 23 15:53:11 2020 +0000 modified: Externals.cfg and sorc/checkout.sh to check out model tag GFS.v16.0.7 and UPP tag upp_gfsv16_release.v1.0.9. Changes include: 1) Inline POST Issues #136 and $142 * Update ceiling height calculation for global FV3. * add low,middle,high instantaneous cloud fraction * add radar reflectivity at model layers 1 and 2 , and radar reflectivities at 1 and 4-km height. * fix a bug in initializing DBZI * output mixed layer CAPE/CIN * remove simulated GOES-12 brightness temperature. * change the names of time averaged low/mid/high cloud fractions in grib2 files from "TCDC" to "LCDC/MCDC/HCDC", respectively. 2) Model Issue #152 * update in-line post control files * upgrade post library to 8.0.9 for hera and wcoss_dell_p3 commit dd76002425a03905bfc6ef63d3f43a6813814497 Merge: df89cc80 dd599eaa Author: Kate Friedman Date: Tue Jun 23 11:34:40 2020 -0400 Merge pull request #91 from christopherwharrop-noaa/feature/fix_externals Update version of upp in Externals.cfg to be consistent with sorc/che… commit dd599eaa4f379e1eb8fc5e057f7904b0e6290d48 Author: Christopher Harrop Date: Tue Jun 23 15:25:08 2020 +0000 Update version of upp in Externals.cfg to be consistent with sorc/checkout.sh commit df8dc52e36f605199755615e4bac54d22e139486 Merge: ff46607b df89cc80 Author: Kate.Friedman Date: Mon Jun 22 14:41:17 2020 +0000 Issue #5 - Sync merge branch 'feature/gfsv16b' into port2orion * feature/gfsv16b: Update config.resources Issue #85: update DA checkout to point at release/gfsda.v16.0.0 Issue #85: (1) rename "chgresfcst" as "echgres", (2) add chgres variables to env Issue #85: add cfp option to chgresfcst; enable threads with chgresfcst add checkout.sh to test on Dell Commit changes from debugging addition of chgresfcst on hera First draft to add chgresfcst to rocoto workflow commit df89cc800d3c479c132a5e679a2562af91b32f62 Author: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com> Date: Fri Jun 19 18:40:26 2020 -0400 Update config.resources Generalize setting of echgres threads to be maximum permitted on given platform instead of explicitly setting echgres threads on every platform. commit ff8cd28365377f0ceed87ff8b4b9a9b4b6e91368 Merge: 1dd83b81 965ff420 Author: russ.treadon Date: Thu Jun 18 20:07:59 2020 +0000 Issue #1: merge branch 'feature/chgresfcst' at 965ff42 into feature/gfsv16b commit 965ff4203f132c6b032398a7a56494329714247c Author: russ.treadon Date: Thu Jun 18 20:02:23 2020 +0000 Issue #85: update DA checkout to point at release/gfsda.v16.0.0 commit ff46607bccf6d8ccc4f4aa1e171a847a9161ab0b Merge: ed6b2797 1dd83b81 Author: Kate.Friedman Date: Wed Jun 17 17:23:44 2020 +0000 Issue #5 - Sync merge branch 'feature/gfsv16b' into port2orion * feature/gfsv16b: modified: run_gfsmos_master.sh.dell to still set range=both as the default for running the real-time parallel update checkout.sh to switch back to post version upp_gfsv16_release.v1.0.8. 1.0.9 still has issues. Issue #1 - sync Externals.cfg with checkout.sh update for FV3 GFSv16.0.6 tag modified: scripts/run_gfsmos_master.sh.dell to set default verification type o short. modified: sorc/checkout.sh updated to model tag GFS.v16.0.6 to fix contrib issue on HERA Issue #1 - update Externals.cfg to match updates to checkout.sh Update exwave_post_sbs.sh Adapting wavepostsbs for running on Hera commit 71d714119960feed1f0807708cdf85677ab81d73 Author: russ.treadon Date: Tue Jun 16 20:11:33 2020 +0000 Issue #85: (1) rename "chgresfcst" as "echgres", (2) add chgres variables to env commit 1dd83b818e9f2babbd11b762951721cf2a4c415e Author: fanglin.yang Date: Tue Jun 16 16:07:45 2020 +0000 modified: run_gfsmos_master.sh.dell to still set range=both as the default for running the real-time parallel commit eb0e3b4d009047251c535eea6fdc5958e58e68f7 Author: fanglin.yang Date: Tue Jun 16 14:43:29 2020 +0000 update checkout.sh to switch back to post version upp_gfsv16_release.v1.0.8. 1.0.9 still has issues. commit dfc76f0715a29e52e4d7133a8259c492229a5c22 Author: Kate.Friedman Date: Tue Jun 16 13:32:01 2020 +0000 Issue #1 - sync Externals.cfg with checkout.sh update for FV3 GFSv16.0.6 tag commit 7078bb5c680d54297101fae573579be24411a94a Author: fanglin.yang Date: Tue Jun 16 13:22:43 2020 +0000 modified: scripts/run_gfsmos_master.sh.dell to set default verification type o short. modified: sorc/checkout.sh updated to model tag GFS.v16.0.6 to fix contrib issue on HERA commit 15d5bed4119b067905f5a4bf47656557f2211883 Author: russ.treadon Date: Mon Jun 15 20:47:47 2020 +0000 Issue #85: add cfp option to chgresfcst; enable threads with chgresfcst commit d8782697fa698c00daa242a656246cf5f5d9b537 Author: Kate.Friedman Date: Mon Jun 15 16:00:35 2020 +0000 Issue #1 - update Externals.cfg to match updates to checkout.sh commit a2bd621727701e9526feccc7ce8ebd43be31860f Merge: 22b735d3 295cd05f Author: Kate Friedman Date: Mon Jun 15 11:57:14 2020 -0400 Merge pull request #84 from NOAA-EMC/feature/gfsv16b_herawavepost Adapting wavepostsbs for running on Hera commit 295cd05f306e4d70d09880a41fd60fadce74bab3 Author: Jose-Henrique Alves <47567389+ajhenrique@users.noreply.github.com> Date: Mon Jun 15 11:53:13 2020 -0400 Update exwave_post_sbs.sh Removing obsolete nm variable entries commit 58d1139c196f92bd4bb149008a8d61eb6457fb3c Author: henrique.alves Date: Thu Jun 11 02:44:41 2020 +0000 Adapting wavepostsbs for running on Hera commit c330e60197c38acb724cfdf4a30a20417a6618b4 Author: CoryMartin-NOAA Date: Wed Jun 10 21:17:00 2020 +0000 add checkout.sh to test on Dell commit 201609b2d43acd13a08bf1d5ab2251db90a11d32 Author: CoryMartin-NOAA Date: Wed Jun 10 20:51:51 2020 +0000 Commit changes from debugging addition of chgresfcst on hera commit fc3066c2b7a5edd9f0d510b88f8542b07b8a8589 Author: CoryMartin-NOAA Date: Wed Jun 10 14:19:07 2020 +0000 First draft to add chgresfcst to rocoto workflow --- Externals.cfg | 8 +- INFO | 3 + README.md | 14 +- env/HERA.env | 8 +- env/JET.env | 4 + env/WCOSS_C.env | 4 + env/WCOSS_DELL_P3.env | 4 + jobs/rocoto/echgres.sh | 13 ++ parm/config/config.echgres | 13 ++ parm/config/config.resources | 9 +- scripts/exwave_init.sh | 1 + scripts/exwave_post_sbs.sh | 169 +++++++++++----------- scripts/exwave_prep.sh | 1 + sorc/enkf_chgres_recenter_nc.fd/setup.f90 | 6 +- sorc/link_fv3gfs.sh | 2 + ush/rocoto/setup_workflow.py | 47 ++++-- ush/wave_grib2_sbs.sh | 1 + ush/wave_grid_interp.sh | 1 + ush/wave_grid_interp_sbs.sh | 1 + ush/wave_grid_moddef.sh | 3 +- ush/wave_outp_spec.sh | 5 +- ush/wave_prnc_cur.sh | 3 + ush/wave_prnc_ice.sh | 1 + ush/wave_tar.sh | 3 +- 24 files changed, 203 insertions(+), 121 deletions(-) create mode 100755 jobs/rocoto/echgres.sh create mode 100755 parm/config/config.echgres diff --git a/Externals.cfg b/Externals.cfg index abfb59ca55..9de84c66ca 100644 --- a/Externals.cfg +++ b/Externals.cfg @@ -1,8 +1,8 @@ # External sub-modules of global-workflow #JKH[FV3GFS] -#JKHtag = orion_gfs.v16 -#JKHlocal_path = sorc/fv3gfs_emc.fd +#JKHtag = GFS.v16.0.7 +#JKHlocal_path = sorc/fv3gfs.fd #JKHrepo_url = https://github.com/ufs-community/ufs-weather-model.git #JKHprotocol = git #JKHrequired = True @@ -15,7 +15,7 @@ protocol = git required = True [GSI] -tag = release/gfsda.v16.0.0 +branch = release/gfsda.v16.0.0 local_path = sorc/gsi.fd repo_url = https://github.com/NOAA-EMC/GSI.git protocol = git @@ -44,7 +44,7 @@ protocol = git required = True [EMC_verif-global] -tag = verif_global_v1.8.1 +tag = verif_global_v1.8.0 local_path = sorc/verif-global.fd repo_url = https://github.com/NOAA-EMC/EMC_verif-global.git protocol = git diff --git a/INFO b/INFO index e4cc622ab2..180d4dfa93 100644 --- a/INFO +++ b/INFO @@ -1,2 +1,5 @@ +01 July 2020 + - merge 23Jun20 portorion branch, ed6b279, into gsd-ccpp-jet + - additional updates for running on jet 17 June 2020 - created new branch, gsd-ccpp-jet, from 15Jun20 port2orion branch, ed6b279 diff --git a/README.md b/README.md index 3fac2751aa..1d8231e6d1 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,6 @@ The global-workflow depends on the following prerequisities to be available on t * workload management platform / scheduler - LSF or SLURM * workflow manager - ROCOTO (https://github.com/christopherwharrop/rocoto) * modules - NCEPLIBS (various), esmf v8.0.0bs48, hdf5, intel/ips v18, impi v18, wgrib2, netcdf v4.7.0, hpss, gempak (see module files under /modulefiles for additional details) -* manage_externals - A utility from ESMCI to checkout external dependencies. Manage_externals can be obtained at the following address and should be in the users PATH: https://github.com/ESMCI/manage_externals The global-workflow current supports the following machines: @@ -18,20 +17,11 @@ The global-workflow current supports the following machines: ### 1. Check out components -The global-workflow uses the manage_externals utility to handle checking out its components. The manic-v1.1.8 manage_externals tag is supported. The manage_externals utility will be replacing the current checkout.sh script. - -Run manage_externals (checkout_externals) while at top of clone: - +While in /sorc folder: ``` -$ checkout_externals -e Externals.cfg +$ sh checkout.sh ``` -If checkout_externals is not in your $PATH then use full path to it: - -* WCOSS-Dell: /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals -* WCOSS-Cray: /gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals -* Hera: /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.8/checkout_externals - ### 2. Build components While in /sorc folder: diff --git a/env/HERA.env b/env/HERA.env index 1002afd00d..af0dd33074 100755 --- a/env/HERA.env +++ b/env/HERA.env @@ -74,9 +74,15 @@ elif [ $step = "anal" ]; then export NTHREADS_GAUSFCANL=1 npe_gausfcanl=${npe_gausfcanl:-1} - export APRUN_GAUSFCANL="$launcher -n $npe_gausfcanl" + export NTHREADS_CHGRES=${nth_echgres:-1} + [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max + export APRUN_CHGRES="" + + export USE_CFP=${USE_CFP:-"YES"} + export APRUNCFP="$launcher" + elif [ $step = "gldas" ]; then nth_max=$(($npe_node_max / $npe_node_gldas)) diff --git a/env/JET.env b/env/JET.env index 96a3122712..9e52d64d6b 100755 --- a/env/JET.env +++ b/env/JET.env @@ -80,6 +80,10 @@ elif [ $step = "anal" ]; then export APRUN_GAUSFCANL="$launcher -n $npe_gausfcanl" + export NTHREADS_CHGRES=${nth_echgres:-1} + [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max + export APRUN_CHGRES="" + elif [ $step = "gldas" ]; then nth_max=$(($npe_node_max / $npe_node_gldas)) diff --git a/env/WCOSS_C.env b/env/WCOSS_C.env index b6a40b977d..8be3f66fe3 100755 --- a/env/WCOSS_C.env +++ b/env/WCOSS_C.env @@ -66,6 +66,10 @@ elif [ $step = "anal" ]; then [[ $npe_node_gausfcanl -gt $npe_gausfcanl ]] && npe_node_gausfcanl=$npe_gausfcanl export APRUN_GAUSFCANL="$launcher -j 1 -n $npe_gausfcanl -N $npe_node_gausfcanl -d $NTHREADS_GAUSFCANL -cc depth" + export NTHREADS_CHGRES=${nth_echgres:-1} + [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max + export APRUN_CHGRES="$launcher -j 1 -n 1 -N 1 -d $NTHREADS_CHGRES -cc depth" + elif [ $step = "gldas" ]; then nth_max=$(($npe_node_max / $npe_node_gldas)) diff --git a/env/WCOSS_DELL_P3.env b/env/WCOSS_DELL_P3.env index 484b5ce73c..c12ec6a327 100755 --- a/env/WCOSS_DELL_P3.env +++ b/env/WCOSS_DELL_P3.env @@ -66,6 +66,10 @@ elif [ $step = "anal" ]; then npe_gausfcanl=${npe_gausfcanl:-1} export APRUN_GAUSFCANL="$launcher $npe_gausfcanl" + export NTHREADS_CHGRES=${nth_echgres:-14} + [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max + export APRUN_CHGRES="" + if [ ${USE_CFP:-"NO"} = "YES" ]; then export APRUNCFP="$launcher \$ncmd $mpmd" fi diff --git a/jobs/rocoto/echgres.sh b/jobs/rocoto/echgres.sh new file mode 100755 index 0000000000..5278f58539 --- /dev/null +++ b/jobs/rocoto/echgres.sh @@ -0,0 +1,13 @@ +#!/bin/ksh -x + +############################################################### +# Source FV3GFS workflow modules +. $HOMEgfs/ush/load_fv3gfs_modules.sh +status=$? +[[ $status -ne 0 ]] && exit $status + +############################################################### +# Execute the JJOB +$HOMEgfs/jobs/JGDAS_ENKF_CHGRES +status=$? +exit $status diff --git a/parm/config/config.echgres b/parm/config/config.echgres new file mode 100755 index 0000000000..c7b2061545 --- /dev/null +++ b/parm/config/config.echgres @@ -0,0 +1,13 @@ +#!/bin/ksh -x + +########## config.echgres ########## +# regrid full-res forecast for use in ensemble-res analysis generation + +echo "BEGIN: config.echgres" + +# Get task specific resources +. $EXPDIR/config.resources echgres + +export CHGRESFCSTSH=$HOMEgfs/scripts/exglobal_enkf_chgres_fv3gfs.sh.ecf + +echo "END: config.echgres" diff --git a/parm/config/config.resources b/parm/config/config.resources index c87b2b7ee6..19c0cdfbc4 100755 --- a/parm/config/config.resources +++ b/parm/config/config.resources @@ -8,7 +8,7 @@ if [ $# -ne 1 ]; then echo "Must specify an input task argument to set resource variables!" echo "argument can be any one of the following:" - echo "anal analcalc analdiag gldas fcst post vrfy metp arch" + echo "anal analcalc analdiag gldas fcst post vrfy metp arch echgres" echo "eobs ediag eomg eupd ecen esfc efcs epos earc" echo "waveinit waveprep wavepostsbs wavegempaksbs waveawipssbs" echo "wavepost waveawips wavestat" @@ -202,6 +202,13 @@ elif [ $step = "metp" ]; then export memory_metp="16384M" fi +elif [ $step = "echgres" ]; then + + export wtime_echgres="01:00:00" + export npe_echgres=3 + export nth_echgres=$npe_node_max + export npe_node_echgres=1 + elif [ $step = "arch" -o $step = "earc" -o $step = "getic" ]; then eval "export wtime_$step='06:00:00'" diff --git a/scripts/exwave_init.sh b/scripts/exwave_init.sh index 848222d5ed..8cb755ec18 100755 --- a/scripts/exwave_init.sh +++ b/scripts/exwave_init.sh @@ -14,6 +14,7 @@ # Script history log: # 2019-05-06 J-Henrique Alves First Version. # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ # diff --git a/scripts/exwave_post_sbs.sh b/scripts/exwave_post_sbs.sh index 52f55cb074..bb40781188 100755 --- a/scripts/exwave_post_sbs.sh +++ b/scripts/exwave_post_sbs.sh @@ -19,7 +19,8 @@ # wave_tar.sh : tars the spectral and bulletin multiple files # # Script history log: -# 2019-12-06 J-Henrique Alves First Version adapted from HTolman post.sh 2007 +# 2019-12-06 J-Henrique Alves: First Version adapted from HTolman post.sh 2007 +# 2020-06-10 J-Henrique Alves: Porting to R&D machine Hera # # $Id$ # @@ -516,67 +517,7 @@ export GRDIDATA=${DATA}/output_$YMDHMS ln -fs $DATA/mod_def.${waveuoutpGRD} mod_def.ww3 - if [ $fhr = $fhrp ] - then - iwait=0 - pfile=$COMIN/rundata/${WAV_MOD_TAG}.out_pnt.${waveuoutpGRD}.${YMD}.${HMS} - while [ ! -s ${pfile} ]; do sleep 10; ((iwait++)) && ((iwait==$iwaitmax)) && break ; echo $iwait; done - if [ $iwait -eq $iwaitmax ]; then - echo " FATAL ERROR : NO RAW POINT OUTPUT FILE out_pnt.$waveuoutpGRD - echo ' ' - [[ "$LOUD" = YES ]] && set -x - postmsg "$jlogfile" "FATAL ERROR : NO RAW POINT OUTPUT FILE out_pnt.$waveuoutpGRD - err=6; export err;${errchk} - exit $err - fi - ln -fs ${pfile} ./out_pnt.${waveuoutpGRD} - - if [ "$DOSPC_WAV" = 'YES' ] - then - export dtspec=3600. - if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0 ; fi # Counter for MP CFP - for buoy in $buoys - do - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_outp_spec.sh $buoy $ymdh spec > spec_$buoy.out 2>&1" >> ${fcmdnow} - nm=`expr $nm + 1` - else - echo "$USHwave/wave_outp_spec.sh $buoy $ymdh spec > spec_$buoy.out 2>&1" >> ${fcmdnow} - fi - done - fi - - if [ "$DOIBP_WAV" = 'YES' ] - then - export dtspec=3600. - if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0 ; fi # Counter for MP CFP - for buoy in $ibpoints - do - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_outp_spec.sh $buoy $ymdh ibp > ibp_$buoy.out 2>&1" >> ${fcmdnow} - nm=`expr $nm + 1` - else - echo "$USHwave/wave_outp_spec.sh $buoy $ymdh ibp > ibp_$buoy.out 2>&1" >> ${fcmdnow} - fi - done - fi - - if [ "$DOBLL_WAV" = 'YES' ] - then - export dtspec=3600. - if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0 ; fi # Counter for MP CFP - for buoy in $buoys - do - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_outp_spec.sh $buoy $ymdh bull > bull_$buoy.out 2>&1" >> ${fcmdnow} - nm=`expr $nm + 1` - else - echo "$USHwave/wave_outp_spec.sh $buoy $ymdh bull > bull_$buoy.out 2>&1" >> ${fcmdnow} - fi - done - fi - - fi +# Gridded data (main part, need to be run side-by-side with forecast if [ $fhr = $fhrg ] then @@ -607,11 +548,7 @@ glo_15mxt) ymdh_int=`$NDATE -${WAVHINDH} $ymdh`; dt_int=3600.; n_int=9999 ;; glo_30mxt) ymdh_int=`$NDATE -${WAVHINDH} $ymdh`; dt_int=3600.; n_int=9999 ;; esac - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_grid_interp_sbs.sh $grdID $ymdh_int $dt_int $n_int > grint_$grdID.out 2>&1" >> ${fcmdigrd}.${nigrd} - else echo "$USHwave/wave_grid_interp_sbs.sh $grdID $ymdh_int $dt_int $n_int > grint_$grdID.out 2>&1" >> ${fcmdigrd}.${nigrd} - fi if [ "$DOGRB_WAV" = 'YES' ] then gribFL=\'`echo ${OUTPARS_WAV}`\' @@ -619,13 +556,10 @@ glo_15mxt) GRDNAME='global' ; GRDRES=0p25 ; GRIDNR=255 ; MODNR=11 ;; glo_30mxt) GRDNAME='global' ; GRDRES=0p50 ; GRIDNR=255 ; MODNR=11 ;; esac - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_grib2_sbs.sh $grdID $GRIDNR $MODNR $ymdh $fhr $GRDNAME $GRDRES $gribFL > grib_$grdID.out 2>&1" >> ${fcmdigrd}.${nigrd} - else echo "$USHwave/wave_grib2_sbs.sh $grdID $GRIDNR $MODNR $ymdh $fhr $GRDNAME $GRDRES $gribFL > grib_$grdID.out 2>&1" >> ${fcmdigrd}.${nigrd} - fi fi - echo "${fcmdigrd}.${nigrd}" >> ${fcmdnow} + echo "pwd" >> ${fcmdnow} + echo "${GRIBDATA}/${fcmdigrd}.${nigrd}" >> ${fcmdnow} chmod 744 ${fcmdigrd}.${nigrd} nigrd=$((nigrd+1)) if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=`expr $nm + 1` ; fi # Increment counter for fcmdigrd @@ -649,17 +583,84 @@ so_20m) GRDNAME='antarc' ; GRDRES=0p33 ; GRIDNR=255 ; MODNR=11 ;; glo_15mxt) GRDNAME='global' ; GRDRES=0p25 ; GRIDNR=255 ; MODNR=11 ;; esac - if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "$nm $USHwave/wave_grib2_sbs.sh $grdID $GRIDNR $MODNR $ymdh $fhr $GRDNAME $GRDRES $gribFL > grib_$grdID.out 2>&1" >> ${fcmdnow} - nm=`expr $nm + 1` - else echo "$USHwave/wave_grib2_sbs.sh $grdID $GRIDNR $MODNR $ymdh $fhr $GRDNAME $GRDRES $gribFL > grib_$grdID.out 2>&1" >> ${fcmdnow} - fi done fi fi +# Point output part (can be split or become meta-task to reduce resource usage) + if [ $fhr = $fhrp ] + then + iwait=0 + pfile=$COMIN/rundata/${WAV_MOD_TAG}.out_pnt.${waveuoutpGRD}.${YMD}.${HMS} + while [ ! -s ${pfile} ]; do sleep 10; ((iwait++)) && ((iwait==$iwaitmax)) && break ; echo $iwait; done + if [ $iwait -eq $iwaitmax ]; then + echo " FATAL ERROR : NO RAW POINT OUTPUT FILE out_pnt.$waveuoutpGRD + echo ' ' + [[ "$LOUD" = YES ]] && set -x + postmsg "$jlogfile" "FATAL ERROR : NO RAW POINT OUTPUT FILE out_pnt.$waveuoutpGRD + err=6; export err;${errchk} + exit $err + fi + ln -fs ${pfile} ./out_pnt.${waveuoutpGRD} + + if [ "$DOSPC_WAV" = 'YES' ] + then + export dtspec=3600. + for buoy in $buoys + do + echo "$USHwave/wave_outp_spec.sh $buoy $ymdh spec > spec_$buoy.out 2>&1" >> ${fcmdnow} + done + fi + + if [ "$DOIBP_WAV" = 'YES' ] + then + export dtspec=3600. + for buoy in $ibpoints + do + echo "$USHwave/wave_outp_spec.sh $buoy $ymdh ibp > ibp_$buoy.out 2>&1" >> ${fcmdnow} + done + fi + + if [ "$DOBLL_WAV" = 'YES' ] + then + export dtspec=3600. + for buoy in $buoys + do + echo "$USHwave/wave_outp_spec.sh $buoy $ymdh bull > bull_$buoy.out 2>&1" >> ${fcmdnow} + done + fi + + fi + + if [ ${CFP_MP:-"NO"} = "YES" ]; then + nfile=0 + ifile=0 + iline=1 + ifirst='yes' + nlines=$( wc -l ${fcmdnow} | awk '{print $1}' ) + while [ $iline -le $nlines ]; do + line=$( sed -n ''$iline'p' ${fcmdnow} ) + if [ -z "$line" ]; then + break + else + if [ "$ifirst" = 'yes' ]; then + echo "#!/bin/sh" > cmdmfile.$nfile + echo "$nfile cmdmfile.$nfile" >> cmdmprog + chmod 744 cmdmfile.$nfile + fi + echo $line >> cmdmfile.$nfile + nfile=$(( nfile + 1 )) + if [ $nfile -eq $NTASKS ]; then + nfile=0 + ifirst='no' + fi + iline=$(( iline + 1 )) + fi + done + fi + wavenproc=`wc -l ${fcmdnow} | awk '{print $1}'` wavenproc=`echo $((${wavenproc}<${NTASKS}?${wavenproc}:${NTASKS}))` @@ -673,7 +674,7 @@ if [ "$wavenproc" -gt '1' ] then if [ ${CFP_MP:-"NO"} = "YES" ]; then - ${wavempexec} -n ${wavenproc} ${wave_mpmd} ${fcmdnow} + ${wavempexec} -n ${wavenproc} ${wave_mpmd} cmdmprog else ${wavempexec} ${wavenproc} ${wave_mpmd} ${fcmdnow} fi @@ -738,10 +739,13 @@ # 6.b Spectral data files + if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0; fi + if [ "$DOIBP_WAV" = 'YES' ] then if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "0 $USHwave/wave_tar.sh $WAV_MOD_TAG ibp $Nibp > ${WAV_MOD_TAG}_ibp_tar.out 2>&1 " >> cmdtarfile + echo "$nm $USHwave/wave_tar.sh $WAV_MOD_TAG ibp $Nibp > ${WAV_MOD_TAG}_ibp_tar.out 2>&1 " >> cmdtarfile + nm=$(( nm + 1 )) else echo "$USHwave/wave_tar.sh $WAV_MOD_TAG ibp $Nibp > ${WAV_MOD_TAG}_ibp_tar.out 2>&1 " >> cmdtarfile fi @@ -749,9 +753,12 @@ if [ "$DOSPC_WAV" = 'YES' ] then if [ ${CFP_MP:-"NO"} = "YES" ]; then - echo "1 $USHwave/wave_tar.sh $WAV_MOD_TAG spec $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile - echo "2 $USHwave/wave_tar.sh $WAV_MOD_TAG bull $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile - echo "3 $USHwave/wave_tar.sh $WAV_MOD_TAG cbull $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile + echo "$nm $USHwave/wave_tar.sh $WAV_MOD_TAG spec $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile + nm=$(( nm + 1 )) + echo "$nm $USHwave/wave_tar.sh $WAV_MOD_TAG bull $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile + nm=$(( nm + 1 )) + echo "$nm $USHwave/wave_tar.sh $WAV_MOD_TAG cbull $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile + nm=$(( nm + 1 )) else echo "$USHwave/wave_tar.sh $WAV_MOD_TAG spec $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile echo "$USHwave/wave_tar.sh $WAV_MOD_TAG bull $Nb > ${WAV_MOD_TAG}_spec_tar.out 2>&1 " >> cmdtarfile diff --git a/scripts/exwave_prep.sh b/scripts/exwave_prep.sh index 98711b574f..4161943fc8 100755 --- a/scripts/exwave_prep.sh +++ b/scripts/exwave_prep.sh @@ -30,6 +30,7 @@ # Nov2012 JHAlves - Transitioning to WCOSS # # Apr2019 JHAlves - Transitioning to GEFS workflow # # Nov2019 JHAlves - Merging wave scripts to global workflow # +# Jun2020 JHAlves - Porting to R&D machine Hera # # # # WAV_MOD_ID and WAV_MOD_TAG replace modID. WAV_MOD_TAG # # is used for ensemble-specific I/O. For deterministic # diff --git a/sorc/enkf_chgres_recenter_nc.fd/setup.f90 b/sorc/enkf_chgres_recenter_nc.fd/setup.f90 index 4bb1a0253c..1ea5b98a82 100644 --- a/sorc/enkf_chgres_recenter_nc.fd/setup.f90 +++ b/sorc/enkf_chgres_recenter_nc.fd/setup.f90 @@ -23,6 +23,7 @@ subroutine program_setup implicit none integer :: istat + character(len=500) :: filenamelist namelist /chgres_setup/ i_output, j_output, input_file, output_file, & terrain_file, vcoord_file, cld_amt @@ -30,8 +31,9 @@ subroutine program_setup cld_amt = .false. ! default option print* - print*,"OPEN SETUP NAMELIST." - open(43, file="./chgres_nc_gauss.nml", iostat=istat) + call getarg(1,filenamelist) + print*,"OPEN SETUP NAMELIST ",trim(filenamelist) + open(43, file=filenamelist, iostat=istat) if (istat /= 0) then print*,"FATAL ERROR OPENING NAMELIST FILE. ISTAT IS: ",istat stop diff --git a/sorc/link_fv3gfs.sh b/sorc/link_fv3gfs.sh index b8fbd7299a..38278c8613 100755 --- a/sorc/link_fv3gfs.sh +++ b/sorc/link_fv3gfs.sh @@ -118,6 +118,7 @@ cd ${pwd}/../jobs ||exit 8 $LINK ../sorc/gsi.fd/jobs/JGDAS_ENKF_SURFACE . $LINK ../sorc/gsi.fd/jobs/JGDAS_ENKF_FCST . $LINK ../sorc/gsi.fd/jobs/JGDAS_ENKF_POST . + $LINK ../sorc/gsi.fd/jobs/JGDAS_ENKF_CHGRES . cd ${pwd}/../scripts ||exit 8 $LINK ../sorc/gsi.fd/scripts/exglobal_analysis_fv3gfs.sh.ecf . $LINK ../sorc/gsi.fd/scripts/exglobal_analcalc_fv3gfs.sh.ecf . @@ -129,6 +130,7 @@ cd ${pwd}/../scripts ||exit 8 $LINK ../sorc/gsi.fd/scripts/exglobal_enkf_surface_fv3gfs.sh.ecf . $LINK ../sorc/gsi.fd/scripts/exglobal_enkf_fcst_fv3gfs.sh.ecf . $LINK ../sorc/gsi.fd/scripts/exglobal_enkf_post_fv3gfs.sh.ecf . + $LINK ../sorc/gsi.fd/scripts/exglobal_enkf_chgres_fv3gfs.sh.ecf . cd ${pwd}/../fix ||exit 8 [[ -d fix_gsi ]] && rm -rf fix_gsi $LINK ../sorc/gsi.fd/fix fix_gsi diff --git a/ush/rocoto/setup_workflow.py b/ush/rocoto/setup_workflow.py index ed93f5043c..2cc4cab1a5 100755 --- a/ush/rocoto/setup_workflow.py +++ b/ush/rocoto/setup_workflow.py @@ -54,7 +54,7 @@ def main(): #wav_steps_awips = ['waveawipssbs', 'waveawips'] # From gfsv16b latest # gfs_steps = ['prep', 'anal', 'gldas', 'fcst', 'postsnd', 'post', 'awips', 'gempak', 'vrfy', 'metp', 'arch'] - hyb_steps = ['eobs', 'ediag', 'eomg', 'eupd', 'ecen', 'esfc', 'efcs', 'epos', 'earc'] + hyb_steps = ['eobs', 'ediag', 'eomg', 'eupd', 'ecen', 'esfc', 'efcs', 'echgres', 'epos', 'earc'] steps = gfs_steps + hyb_steps if _base.get('DOHYBVAR', 'NO') == 'YES' else gfs_steps steps = steps + metp_steps if _base.get('DO_METP', 'NO') == 'YES' else steps @@ -312,9 +312,9 @@ def get_hyb_resources(dict_configs): # These tasks can be run in either or both cycles if lobsdiag_forenkf in ['.T.', '.TRUE.']: - tasks1 = ['eobs', 'ediag', 'eupd'] + tasks1 = ['eobs', 'ediag', 'eupd', 'echgres'] else: - tasks1 = ['eobs', 'eomg', 'eupd'] + tasks1 = ['eobs', 'eomg', 'eupd', 'echgres'] if eupd_cyc in ['BOTH']: cdumps = ['gfs', 'gdas'] @@ -478,18 +478,14 @@ def get_gdasgfs_tasks(dict_configs, cdump='gdas'): dep_dict = {'type': 'data', 'data': data} deps1.append(rocoto.add_dependency(dep_dict)) dep_dict = {'type': 'task', 'name': '%sanal' % cdump} - deps1.append(rocoto.add_dependency(dep_dict)) - dependencies1 = rocoto.create_dependency(dep_condition='or', dep=deps1) - - if cdump in ['gdas']: - deps2 = [] - deps2 = dependencies1 - dep_dict = {'type': 'cycleexist', 'offset': '-06:00:00'} - deps2.append(rocoto.add_dependency(dep_dict)) - dependencies2 = rocoto.create_dependency(dep_condition='and', dep=deps2) - task = wfu.create_wf_task('analcalc', cdump=cdump, envar=envars, dependency=dependencies2) + deps.append(rocoto.add_dependency(dep_dict)) + if dohybvar in ['y', 'Y', 'yes', 'YES'] and cdump == 'gdas': + dep_dict = {'type': 'task', 'name': '%sechgres' % 'gdas', 'offset': '-06:00:00'} + deps.append(rocoto.add_dependency(dep_dict)) + dependencies = rocoto.create_dependency(dep_condition='and', dep=deps) else: - task = wfu.create_wf_task('analcalc', cdump=cdump, envar=envars, dependency=dependencies1) + dependencies = rocoto.create_dependency(dep_condition='or', dep=deps) + task = wfu.create_wf_task('analcalc', cdump=cdump, envar=envars, dependency=dependencies) dict_tasks['%sanalcalc' % cdump] = task @@ -888,7 +884,6 @@ def get_hyb_tasks(dict_configs, cycledef='enkf'): dict_tasks['%sesfc' % cdump] = task - # efmn, efcs deps1 = [] dep_dict = {'type': 'metatask', 'name': '%secmn' % cdump} @@ -909,6 +904,17 @@ def get_hyb_tasks(dict_configs, cycledef='enkf'): dict_tasks['%sefmn' % cdump] = task + # echgres + deps1 = [] + dep_dict = {'type': 'task', 'name': '%sfcst' % cdump} + deps1.append(rocoto.add_dependency(dep_dict)) + dep_dict = {'type': 'metatask', 'name': '%sefmn' % cdump} + deps1.append(rocoto.add_dependency(dep_dict)) + dependencies1 = rocoto.create_dependency(dep_condition='and', dep=deps1) + task = wfu.create_wf_task('echgres', cdump=cdump, envar=envars1, dependency=dependencies1, cycledef=cycledef) + + dict_tasks['%sechgres' % cdump] = task + # epmn, epos deps = [] dep_dict = {'type': 'metatask', 'name': '%sefmn' % cdump} @@ -1131,7 +1137,16 @@ def create_xml(dict_configs): dict_hyb_tasks = get_hyb_tasks(dict_configs) # Removes &MEMORY_JOB_DUMP post mortem from hyb tasks - hyp_tasks = {'gdaseobs':'gdaseobs', 'gdasediag':'gdasediag', 'gdaseomg':'gdaseomn', 'gdaseupd':'gdaseupd','gdasecen':'gdasecmn','gdasesfc':'gdasesfc','gdasefcs':'gdasefmn','gdasepos':'gdasepmn','gdasearc':'gdaseamn'} + hyp_tasks = {'gdaseobs':'gdaseobs', + 'gdasediag':'gdasediag', + 'gdaseomg':'gdaseomn', + 'gdaseupd':'gdaseupd', + 'gdasecen':'gdasecmn', + 'gdasesfc':'gdasesfc', + 'gdasefcs':'gdasefmn', + 'gdasepos':'gdasepmn', + 'gdasearc':'gdaseamn', + 'gdasechgres':'gdasechgres'} for each_task, each_resource_string in dict_hyb_resources.iteritems(): #print each_task,hyp_tasks[each_task] #print dict_hyb_tasks[hyp_tasks[each_task]] diff --git a/ush/wave_grib2_sbs.sh b/ush/wave_grib2_sbs.sh index 2afec2c65e..0c04f0fcb2 100755 --- a/ush/wave_grib2_sbs.sh +++ b/ush/wave_grib2_sbs.sh @@ -11,6 +11,7 @@ # # Script history log: # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ # diff --git a/ush/wave_grid_interp.sh b/ush/wave_grid_interp.sh index 273d8a813d..0d90b9c56c 100755 --- a/ush/wave_grid_interp.sh +++ b/ush/wave_grid_interp.sh @@ -11,6 +11,7 @@ # # Script history log: # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ ############################################################################### diff --git a/ush/wave_grid_interp_sbs.sh b/ush/wave_grid_interp_sbs.sh index f2e0855331..d810564f9f 100755 --- a/ush/wave_grid_interp_sbs.sh +++ b/ush/wave_grid_interp_sbs.sh @@ -11,6 +11,7 @@ # # Script history log: # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ # diff --git a/ush/wave_grid_moddef.sh b/ush/wave_grid_moddef.sh index 2b82e8570e..9eb41aaba1 100755 --- a/ush/wave_grid_moddef.sh +++ b/ush/wave_grid_moddef.sh @@ -11,6 +11,7 @@ # # Script history log: # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ # # @@ -126,7 +127,7 @@ # 3. Clean up cd .. - #rm -rf moddef_$grdID + rm -rf moddef_$grdID set +x echo ' ' diff --git a/ush/wave_outp_spec.sh b/ush/wave_outp_spec.sh index 2e902bc212..5ec718efab 100755 --- a/ush/wave_outp_spec.sh +++ b/ush/wave_outp_spec.sh @@ -11,6 +11,7 @@ # # Script history log: # 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported to R&D machine Hera # # $Id$ # @@ -245,8 +246,8 @@ # 3.b Clean up the rest -# rm -f ww3_outp.inp -# rm -f mod_def.ww3 out_pnt.ww3 + rm -f ww3_outp.inp + rm -f mod_def.ww3 out_pnt.ww3 cd .. rm -rf ${specdir}_${bloc} diff --git a/ush/wave_prnc_cur.sh b/ush/wave_prnc_cur.sh index 848088540c..4c731e5ddd 100755 --- a/ush/wave_prnc_cur.sh +++ b/ush/wave_prnc_cur.sh @@ -10,6 +10,9 @@ # Abstract: Creates current binary data for forcing WW3 # # Script history log: +# 2019-10-02 J-Henrique Alves: origination, first version +# 2019-11-02 J-Henrique Alves Ported to global-workflow. +# 2020-06-10 J-Henrique Alves Ported R&D machine Hera # # $Id$ # diff --git a/ush/wave_prnc_ice.sh b/ush/wave_prnc_ice.sh index c7a0a1a720..fcc1cab0c9 100755 --- a/ush/wave_prnc_ice.sh +++ b/ush/wave_prnc_ice.sh @@ -16,6 +16,7 @@ # Update record : # # # # - Origination: Hendrik Tolman 01-Mar-2007 # +# 2019-11-02 J-Henrique Alves Ported to global-workflow. # # # Update log # # Nov2019 JHAlves - Merging wave scripts to global workflow # diff --git a/ush/wave_tar.sh b/ush/wave_tar.sh index b5550b6fd4..e367a01414 100755 --- a/ush/wave_tar.sh +++ b/ush/wave_tar.sh @@ -17,6 +17,7 @@ # Origination: Hendrik Tolman March 13, 2007 # # Update log # # Nov2019 JHAlves - Merging wave scripts to global workflow # +# 2020-06-10 J-Henrique Alves Ported R&D machine Hera # # ############################################################################### # @@ -220,7 +221,7 @@ cd $DATA set +x; [[ "$LOUD" = YES ]] && set -v - rm -f ${STA_DIR}/${type} + rm -rf ${STA_DIR}/${type} set +v echo ' '