Skip to content

Commit

Permalink
updates to use CMEPS for coupled-crow (#120)
Browse files Browse the repository at this point in the history
* Pointing to v3.1 tag for ufs-s2s-model
making readme consistant with user.yaml.default

* bug fix, POSTGRB2TBL undefined

* Squashing commits to wave2global

* Fix to JWAVE_PREP to look back a day for rtofs

* Moving standalone fv3 model_config exglobal_fcst block into if/else/fi cplwav model_config block. Reinstating config.wave block in JGLOBAL_FORECAST. Pointing EXECwave to HOMEgfs/exec directory for WW3 util executables (changed link_fv3gfs.sh accordingly). Removing debug options from compile.sh line in build_fv3.sh.

* Clean up exwave_post_sbs

* clean up unused schema
move default value into defaults/

* move all default values into /defaults from /schema

It's better if we have one place to manage default values
for configurable variables other than two

Default values only for reference.

Please throughly test to make sure it could reproduce.

* model_configure_DATM

* merge in the latest revision from IPD work

* add stoch variables to config.fcst
scripts update: CCPP and IPD works

* capital case CDUMP values in config.fv3

* stochy variable namelist update,
add hard-coded variables into config.fcst (fcst.yaml)

* OCNTIM and ICETIM are included in config.fv3
ice variable istep0 bug fix

* stochastic variables update

* delete falseful copy after namelist and model_configure parsing
add FNMSKH conditional clause, coupled vs non-coupled

* update on sandbox platform, for FIX_SCRUB settings

* turn off stochastic features in case file

* configuration update for CCPP scripts

* generalize fix file directory for imp_physics 8

* reiflag default value changed to 1

* port to Orion

* remove HERA-like paths in platform.yaml of Orion

* checkout and External.cfg update

* update checkout and External.cfg

* case file adjustment for Orion IC

* expand gitignore

* automate fix file linking, to this location:
/work/noaa/marine/jmeixner/tempFixICdir/fix/fix_prep_benchmark3

* updates to checkout, build and link to run prototype 4

* update UPP version to v1.0.8

* making Externals_coupled.cfg consistent with checkout.sh

* removing fix from .gitignore

* fixed error when linking on hera

* replace MOM6IC with BASE_CPLIC for base directory location of
IC for s2s model and made it platform dependent.
(Orion location is temporary and needs to be replaced, can also be
changed in cases/coupled_free_forecast.yaml to users prefered location)

* updates for copying ICs

* more updates for ICs

* updates for new fix (which includes prep for multiple resolutions) and some clean-up

* renaming exe to reflect updated build

* update diag_table for ocean

* updates for ice_in to match prototype 4

* updates for CCPP suite used in p4

* making a fix dir of all available ccpp suite files
copying suite xml file to rundirectory

* updating for reorganized tar files, pulling in changes from wcoss-p2 workflow

* last of archive updates for p4

* updating module files to match s2s model and ncep post modules

* we don't want to cycle/have 4 runs per day so setting the cases to be just 1 IC

* updating module files on hera

* updates to readme for orion

* mediator cold start succeeds on hera and forecast job starts
post jobs for atmosphere are no longer triggered on cold starts

* adding ncks, updating ocean post, some fixes for tar files

* updates for tar files and do not keep data

* fix typo

* updating module_base.orion that allows cold start to run.
Not sure if other jobs will now fail though or if what is in your
.bashrc file will effect the success/failure of the run.

* both forecast and atm post seem to be running okay on orion

* wave configuration files

* wave configuration update

* fix module-setup.sh.inc script, which causes module purge unsuccessful

* 1, Moved some of the variables into configuration system from JWAVE scripts
2, Wave configuration system built out
3, Modularized scripts updated to accomodate wave variables

* Fix suite name of CCPP
Move cplwav and cplwav2atm variable

* Revise CCPP_SUITE variable name, default option, to "IPD"

* wave post-process job update

* reg2grb2 for test on feature coupled-crow

* orion hera build test

* reg2grb2 Build on Hera and Orion

* Merged and ready for test

* add ocn-ice build to readme

* adding updates to wave scripts from gfsv16b branch by @ajhenrique

* cleaning up double cplwav

* changes to wave case

* bugfix in layout file

* bug fix in layout file

* move OCNPETS, ICEPETS and WAVEPETS into dedicated model-specific
sessions instead of fv3_gfs_settings

* first try for adding wave ICs

* fix bugs in config.wave and coupled_ic.sh

* updates for wave defaults for s2s p4

* updates for running wave jobs

* point WW3-related executables through HOMEgfs/fv3_coupled.fd/exec

* ocean post hours at boundary are processed twice, now fixed.

* wave init job now works on hera

* adding missing file from last commit

* add ocean, ice and wave configurable variables

* add ocean, ice and wave configurable variables

* put ic type variable into config.fv3ic

* wave init now works on orion

* updates to run wave prep for s2s case

* updating forecast job for wave jobs

* updates for forecast with waves

* at this point the ocean post is now running

* fix nems.configure for cold start

* updated wave-realted scripts to match gfsv16 branch
updated environment files env/ORION.env
updated configuration system

* add waves to jglobal forecast config

* updates - everything but wave post seems to run now, part of wave post does
updating model to latest tag

* change for lmod -> lmod/lmod on orion
updated README

* fixing syntax error in JWAVE_PREP job for variables that are not
actively being used

* adding cycle definition to wave post job

* cleanup from duplications

* updating build to work with CMEPS

* updates for wave post jobs - changing exe dir location

* updates of an lmod->lmod/lmod
update to build of fv3_coupled

* reg2grb2 now builds on orion

* add note to use rocoto/1.3.2 on orion as this version has fix for
the UNAVAILABLE issue

* adding nems.configure to the log file

* updating ice_in see NOAA-EMC/CICE5#25

* updates for nems.configure for running with CMEPS

* adding MED pets so that they only use the forecast not write grid pets

* updates for CMEPS mediator

* add pointer to which mediator restart for cmeps

* final updates to make cmeps forecasts run

* corrections on E-P calculation in regdiag.f

* (1) set IC dir path as /work/noaa/marine/Partha.Bhattacharjee/IC_Dir in workflow/platforms/orion.yaml
(2) add extra white space in ush/nems_configure.sh in line 56 to eliminate warning in run log
(3) correction on restart_n template variable setting in ush/nems.configure.med_atm_ocn_ice.IN

* cleanup

* updating MOM6 for restart parameter to be false

* add extra two ice variables in ush/parsing_namelists_CICE.sh

Co-authored-by: Jessica.Meixner <Jessica.Meixner@noaa.gov>
Co-authored-by: jian.kuang <Jian.Kuang@noaa.gov>
Co-authored-by: Henrique Alves <henrique.alves@noaa.gov>
Co-authored-by: kate.friedman <kate.friedman@noaa.gov>
Co-authored-by: Jose-Henrique Alves <47567389+ajhenrique@users.noreply.github.com>
Co-authored-by: Lin.Gan <lin.gan@noaa.gov>
Co-authored-by: JianKuang-NOAA <51758200+JianKuang-NOAA@users.noreply.github.com>
  • Loading branch information
8 people committed Aug 31, 2020
1 parent c51810c commit f3603c5
Show file tree
Hide file tree
Showing 27 changed files with 342 additions and 322 deletions.
2 changes: 1 addition & 1 deletion Externals_coupled.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[ufs-s2s-model]
tag = s2s_prototype4.0
tag = ufss2s_cmeps_v0.7
local_path = sorc/fv3_coupled.fd
repo_url = https://github.com/ufs-community/ufs-s2s-model
protocol = git
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ or
For Orion:
First make sure you have python loaded:
module load contrib
module load rocoto
module load rocoto #Make sure to use 1.3.2
module load intelpython3
and then replace ORION with HERA in the commands above.
Expand Down
5 changes: 4 additions & 1 deletion jobs/JWAVE_POST_SBS
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@ export COMOUT=${COMOUT:-${COMOUTwave}/${COMPONENTwave}.${PDY}/${cyc}}

mkdir -p $COMOUT/gridded
mkdir -p $COMOUT/station
mkdir -p $COMOUT/stats

# Set mpi serial command
export wavempexec=${launcher:-"mpirun -n"}
export wave_mpmd=${mpmd:-"cfp"}

env | sort

Expand Down
4 changes: 1 addition & 3 deletions modulefiles/modulefile.reg2grb2.orion
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,11 @@
# reg2grb2 - orion
#############################################################
export FCOMP=ifort
module load intel/2020
module load intel/2018.4
module load netcdf/4.7.2
module load ip/3.0.2
module load sp/2.0.3
module load w3nco/2.0.7
module load bacio/2.0.3
module load landsfcutil/2.2.0
module use -a /apps/contrib/NCEPLIBS/orion/modulefiles
module load wgrib/2.0.8
export WGRIB2_ROOT=/apps/contrib/NCEPLIBS/orion/NCEPLIBS/NCEPLIBS-external/build5/wgrib2/grib2
1 change: 1 addition & 0 deletions parm/mom6/MOM_input_025
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ DTFREEZE_DP = -7.75E-08 ! [deg C Pa-1] default = 0.0
! temperature with pressure.

! === module MOM_restart ===
RESTART_CHECKSUMS_REQUIRED = False
PARALLEL_RESTARTFILES = True ! [Boolean] default = False
! If true, each processor writes its own restart file,
! otherwise a single restart file is generated
Expand Down
1 change: 1 addition & 0 deletions parm/mom6/MOM_input_025_wav
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ DTFREEZE_DP = -7.75E-08 ! [deg C Pa-1] default = 0.0
! temperature with pressure.

! === module MOM_restart ===
RESTART_CHECKSUMS_REQUIRED = False
PARALLEL_RESTARTFILES = True ! [Boolean] default = False
! If true, each processor writes its own restart file,
! otherwise a single restart file is generated
Expand Down
11 changes: 8 additions & 3 deletions sorc/build_fv3_coupled.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@ if [ ! -d "../exec" ]; then
mkdir ../exec
fi

if [ $target = hera ]; then target=hera.intel ; fi
if [ $target = orion ]; then target=orion.intel ; fi

cd fv3_coupled.fd/
FV3=$( pwd -P )/FV3
cd tests/
./compile.sh "$FV3" "$target" "CCPP=Y SUITES=FV3_GFS_2017_coupled,FV3_GFS_2017_satmedmf_coupled,FV3_GFS_v15p2_coupled MOM6=Y CICE=Y WW3=Y CMEPS=Y" 1
mv -f fcst_1.exe ../NEMS/exe/nems_fv3_ccpp_mom6_cice5_ww3.x

cd fv3_coupled.fd/NEMS
make -j 8 app=coupledFV3_CCPP_MOM6_CICE_WW3 build
mv ./exe/NEMS.x ./exe/nems_fv3_ccpp_mom6_cice5_ww3.x
4 changes: 2 additions & 2 deletions sorc/machine-setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ if [[ -d /work ]] ; then
# We are on MSU Orion
if ( ! eval module help > /dev/null 2>&1 ) ; then
echo load the module command 1>&2
source /apps/lmod/init/$__ms_shell
source /apps/lmod/lmod/init/$__ms_shell
fi
target=orion
module purge
module load intel/2018.4
module load impi/2018.4
export NCEPLIBS=/apps/contrib/NCEPLIBS/orion
export WRFPATH=$NCEPLIBS/wrf.shared.new/v1.1.1/src
#export WRFPATH=$NCEPLIBS/wrf.shared.new/v1.1.1/src
module use $NCEPLIBS/modulefiles
export myFC=mpiifort
export FCOMP=mpiifort
Expand Down
7 changes: 4 additions & 3 deletions sorc/reg2grb2.fd/regdiag.f
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,7 @@ subroutine diag(im,jm,km,icefile,ocnfile, &
! check why fac=-8.64e6 for evap minus precip
!-------------------------------------------------------------------
data fac/ 0.0, 0.001,1.0,1.0, 1.0, 1.0, 1.0,1.0,1.0,1.0, &
1.0,273.15,1.0,1.0,-8.64e6, 1.0, 1.0,1.0,1.0,1.0, &
1.0,273.15,1.0,1.0,-8.64e3, 1.0, 1.0,1.0,1.0,1.0, &
1.0, 1.0,1.0,1.0, 1.0, 1.0, 1.0,1.0,1.0,1.0, &
1.0, 1.0,1.0,1.0, 1.0, 1.0, 1.0/

Expand Down Expand Up @@ -784,10 +784,11 @@ subroutine diag(im,jm,km,icefile,ocnfile, &
endif
! pme
if (nv.eq.15) then
varsfc=pme
varsfc=lprec-evap !pme
do j=1,jm
do i=1,im
if (varsfc(i,j) .LE. spv_pme) varsfc(i,j)=undef
! if (varsfc(i,j) .LE. spv_pme) varsfc(i,j)=undef
if (lprec(i,j) .LE. spv_pme) varsfc(i,j)=undef
enddo
enddo
endif
Expand Down
6 changes: 4 additions & 2 deletions ush/forecast_postdet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -598,7 +598,8 @@ MOM6_postdet()

# Copy mediator restart files to RUNDIR
if [ $inistep = 'restart' ]; then
$NCP $ROTDIR/$CDUMP.$PDY/$cyc/mediator_* $DATA/
$NCP $ROTDIR/$CDUMP.$PDY/$cyc/ufs.s2s*.nc $DATA/
$NCP $ROTDIR/$CDUMP.$PDY/$cyc/rpointer.cpl $DATA/
fi

echo "SUB ${FUNCNAME[0]}: MOM6 input data linked/copied"
Expand Down Expand Up @@ -629,7 +630,8 @@ MOM6_out()
[[ ! -d $COMOUT ]] && mkdir -m 775 -p $COMOUT

if [ $inistep = 'cold' ]; then
cp $DATA/mediator_* $COMOUT/
cp $DATA/ufs.s2s.cold.cpl.r.*.nc $COMOUT/
cp $DATA/rpointer.cpl $COMOUT/
status=$?
exit $status
else
Expand Down
84 changes: 57 additions & 27 deletions ush/nems.configure.med_atm_ocn_ice.IN
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,6 @@ EARTH_attributes::
# MED #
MED_model: @[med_model]
MED_petlist_bounds: @[med_petlist_bounds]
MED_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
DumpRHs = false
coldstart = @[coldstart]
restart_interval = 0
::

# ATM #
Expand All @@ -25,6 +19,8 @@ ATM_petlist_bounds: @[atm_petlist_bounds]
ATM_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
ProfileMemory = false
OverwriteSlice = true
::

# OCN #
Expand All @@ -33,7 +29,9 @@ OCN_petlist_bounds: @[ocn_petlist_bounds]
OCN_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
restart_interval = 0
ProfileMemory = false
OverwriteSlice = true
dbug_flag = 0
::

# ICE #
Expand All @@ -42,27 +40,59 @@ ICE_petlist_bounds: @[ice_petlist_bounds]
ICE_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
ProfileMemory = false
OverwriteSlice = true
dbug_flag = 0
::

# Run Sequence #
# CMEPS warm run sequence
runSeq::
@@[CPL_SLOW]
MED MedPhase_prep_ocn
MED -> OCN :remapMethod=redist
OCN
@@[CPL_FAST]
MED MedPhase_prep_ice
MED MedPhase_prep_atm
MED -> ATM :remapMethod=redist
MED -> ICE :remapMethod=redist
ATM
ICE
ATM -> MED :remapMethod=redist
ICE -> MED :remapMethod=redist
MED MedPhase_atm_ocn_flux
MED MedPhase_accum_fast
@
OCN -> MED :remapMethod=redist
MED MedPhase_write_restart
@
@@[CPL_SLOW]
MED med_phases_prep_ocn_accum_avg
MED -> OCN :remapMethod=redist
OCN
@@[CPL_FAST]
MED med_phases_prep_atm
MED med_phases_prep_ice
MED -> ATM :remapMethod=redist
MED -> ICE :remapMethod=redist
ATM
ICE
ATM -> MED :remapMethod=redist
ICE -> MED :remapMethod=redist
MED med_fraction_set
MED med_phases_prep_ocn_map
MED med_phases_prep_ocn_merge
MED med_phases_prep_ocn_accum_fast
MED med_phases_profile
@
OCN -> MED :remapMethod=redist
MED med_phases_restart_write
@
::

# CMEPS variables

DRIVER_attributes::
mediator_read_restart = .true.
::
MED_attributes::
ATM_model = @[atm_model]
ICE_model = @[ice_model]
OCN_model = @[ocn_model]
history_n = 0
history_option = nhours
history_ymd = -999
coupling_mode = nems_orig
::
ALLCOMP_attributes::
ScalarFieldCount = 2
ScalarFieldIdxGridNX = 1
ScalarFieldIdxGridNY = 2
ScalarFieldName = cpl_scalars
start_type = startup
case_name = ufs.s2s
restart_n = @[restart_interval_hours]
restart_option = nhours
restart_ymd = -999
::
99 changes: 61 additions & 38 deletions ush/nems.configure.med_atm_ocn_ice_wav.IN
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,6 @@ EARTH_attributes::
# MED #
MED_model: @[med_model]
MED_petlist_bounds: @[med_petlist_bounds]
MED_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
DumpRHs = false
coldstart = false
restart_interval = 0
ProfileMemory = false
::

# ATM #
Expand All @@ -27,7 +20,7 @@ ATM_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
ProfileMemory = false
OverwriteSlice = false
OverwriteSlice = true
::

# OCN #
Expand All @@ -36,9 +29,9 @@ OCN_petlist_bounds: @[ocn_petlist_bounds]
OCN_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
restart_interval = 0
ProfileMemory = false
OverwriteSlice = false
OverwriteSlice = true
dbug_flag = 0
::

# ICE #
Expand All @@ -48,42 +41,72 @@ ICE_attributes::
Verbosity = 0
DumpFields = @[DumpFields]
ProfileMemory = false
OverwriteSlice = false
OverwriteSlice = true
dbug_flag = 0
::

# WAV #
# WAV #
WAV_model: @[wav_model]
WAV_petlist_bounds: @[wav_petlist_bounds]
WAV_attributes::
Verbosity = 0
OverwriteSlice = false
::

# Run Sequence #
# CMEPS warm run sequence
runSeq::
@@[CPL_SLOW]
MED MedPhase_prep_ocn
MED -> OCN :remapMethod=redist
OCN -> WAV
WAV -> OCN :srcMaskValues=1
OCN
@@[CPL_FAST]
MED MedPhase_prep_ice
MED MedPhase_prep_atm
MED -> ATM :remapMethod=redist
MED -> ICE :remapMethod=redist
WAV -> ATM :srcMaskValues=1
ATM -> WAV
ICE -> WAV
ATM
ICE
WAV
ATM -> MED :remapMethod=redist
ICE -> MED :remapMethod=redist
MED MedPhase_atm_ocn_flux
MED MedPhase_accum_fast
@
OCN -> MED :remapMethod=redist
MED MedPhase_write_restart
@
@@[CPL_SLOW]
MED med_phases_prep_ocn_accum_avg
MED -> OCN :remapMethod=redist
OCN -> WAV
WAV -> OCN :srcMaskValues=1
OCN
@@[CPL_FAST]
MED med_phases_prep_atm
MED med_phases_prep_ice
MED -> ATM :remapMethod=redist
MED -> ICE :remapMethod=redist
WAV -> ATM :srcMaskValues=1
ATM -> WAV
ICE -> WAV
ATM
ICE
WAV
ATM -> MED :remapMethod=redist
ICE -> MED :remapMethod=redist
MED med_fraction_set
MED med_phases_prep_ocn_map
MED med_phases_prep_ocn_merge
MED med_phases_prep_ocn_accum_fast
MED med_phases_profile
@
OCN -> MED :remapMethod=redist
MED med_phases_restart_write
@
::

# CMEPS variables

DRIVER_attributes::
mediator_read_restart = .true.
::
MED_attributes::
ATM_model = @[atm_model]
ICE_model = @[ice_model]
OCN_model = @[ocn_model]
history_n = 0
history_option = nhours
history_ymd = -999
coupling_mode = nems_orig
::
ALLCOMP_attributes::
ScalarFieldCount = 2
ScalarFieldIdxGridNX = 1
ScalarFieldIdxGridNY = 2
ScalarFieldName = cpl_scalars
start_type = startup
case_name = ufs.s2s
restart_n = @[restart_interval_hours]
restart_option = nhours
restart_ymd = -999
::
Loading

0 comments on commit f3603c5

Please sign in to comment.