EFSOI implementation in global-workflow

[AndrewEichmann-NOAA]

This issue is to track the implementation of EFSOI in global-workflow.

With the opening of the issue, it will pick up documentation of the development from Redmine task https://vlab.ncep.noaa.gov/redmine/issues/55974

Development is in https://github.com/AndrewEichmann-NOAA/global-workflow under feature/EFSOI

[AndrewEichmann-NOAA]

For background, EFSOI runs a parallel GDAS task workflow starting with gdaseupd, called gdaseupdfsoi, followed by gdasecenfsoi, gdasesfcfsoi, gdasefmnfsoi, gdasepmnfsoi, and gdasefsoi (which is specific to EFSOI). All except gdasefsoi are largely the same as the regular gdas tasks, with the following exceptions:

• gdaseupdfsoi runs with the fsoi flag on so that it outputs the osense file and uninflated fields
• gdasefmnfsoi and gdasepmnfsoi run and process 30hr ensemble forecasts, specifically to get the 24h and 30h forecasts

The EFSOI-specific tasks have their own scripts in jobs/rocoto, which call EFSOI-specific scripts in jobs. These in turn call task scripts in the scripts directory. These scripts are derived from scripts in the GSI repository, but are currently in the fork of global-workflow. The goal is to have as little duplication as possible of these scripts, but for now they tend to have their EFSOI-specific versions for ease of development.

[AndrewEichmann-NOAA]

At present, the EFSOI tasks have been run through gdasesfcfsoi, and is currently stuck on gdasefmnfsoi (the forecast). The latest resolved issue involved files necessary for a warm start not being available - since the EFSOI ensemble runs in its own tree (efsoigdas.YYYYMMDD) and is running there for its first cycle in the experiment (unlike the regular enkfgdas), there was no previous cycle to warm start from. At present the task is using the script exglobal_efsoi_forecast.sh (from exglobal_forecast.sh), and the variables prefix and rprefix were set to point to EFSOI ensemble. rprefix is used to point to the previous cycle for the warm start files, which don't exist in this case. To fix this, rprefix was set back to rprefix=enkf$rCDUMP to use the same warm start files as the regular ensemble, and the only change from the original script is this:
(base) -bash-4.2$ diff exglobal_forecast.sh exglobal_efsoi_forecast.sh
202c202
< prefix=enkf$CDUMP

prefix=efsoi$CDUMP
(base) -bash-4.2$

It will have to be determined whether the regular ensemble's warm start files can used this way as a solution.

[AndrewEichmann-NOAA]

Current issue, as emailed to Kate:

For the EFSOI implementation I'm trying to get an ensemble forecast with a set of initial conditions different from the regular GDAS. To do this I'm running in a separate directory tree (efsoigdas instead of enkfgdas). I'm stuck at getting the model to run - it gets to the executable and then stops seemingly without a message:

gfs.x /scratch1/NCEPDEV/stmp2/Andrew.Eichmann/RUNDIRS/hedgehog/2019111706/gdas/efcsfsoi.91157/mem001/.2.657 + OMP_NUM_THREADS=42.657 + export OMP_NUM_THREADS2.657 + srun '--export=ALL' /scratch1/NCEPDEV/stmp2/Andrew.Eichmann/RUNDIRS/hedgehog/2019111706/gdas/efcsfsoi.91157/mem001/global_fv3gfs.x2.661 + 1>& 1 2>& 2

• . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * . * .     PROGRAM NEMS (Andrew.Eichmann) Thu Nov 19 23:00:58 UTC 2020 r845c887776c6 https://github.com/NOAA-EMC/NEMS HAS BEGUN. COMPILED 2020324.00     ORG: NEMSapplication called MPI_Abort(comm=0x84000004, 1) - process 60application called MPI_Abort(comm=0x84000004, 1) - process 180application called MPI_Abort(comm=0x84000004, 1) - process 10application called MPI_Abort(comm=0x84000004, 1) - process 50application called MPI_Abort(comm=0x84000004, 1) - process 70

The standard GDAS ensemble ran without a problem, and I've tried to make this as identical to that as possible, except for running with a different ensemble. Where should I be looking for messages? 

The above log is/scratch1/NCEPDEV/stmp4/Andrew.Eichmann/hedgehog/logs/2019111706/gdasefcsfsoi01.log
and its successful equivalent is /scratch1/NCEPDEV/stmp4/Andrew.Eichmann/hedgehog/logs/2019111706/gdasefcs01.log

The script stack (expected and verified in the log):
/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/AFE-fork/global-workflow/jobs/rocoto/efcsfsoi.sh
/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/AFE-fork/global-workflow/jobs/JGDAS_EFSOI_FCST 
/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/AFE-fork/global-workflow/scripts/exgdas_enkf_fcst.sh
/scratch1/NCEPDEV/da/Andrew.Eichmann/fv3gfs/AFE-fork/global-workflow/scripts/exglobal_efsoi_forecast.sh

The only variable differences from a regular GDAS forecast should be:
export COMOUT="$ROTDIR/efsoi$CDUMP.$PDY/$cyc/$COMPONENT"

and 
export FORECASTSH=$HOMEgfs/scripts/exglobal_efsoi_forecast.sh

in JGDAS_EFSOI_FCST 

and the following:
(base) -bash-4.2$ diff exglobal_forecast.sh exglobal_efsoi_forecast.sh
202c202
<   prefix=enkf$CDUMP

  prefix=efsoi$CDUMP
(base) -bash-4.2$ 

The experiment directory is /scratch1/NCEPDEV/da/Andrew.Eichmann/para_gfs/hedgehog

[KateFriedman-NOAA]

So since the error message is lovely and vague I compared the logs to look for a difference (I suspected resource issue based on the error). I found that $NTASKS_FV3 wasn't set in the failed efsoi job:

0.016 + NTASKS_FV3=''

...which led to PE_MEMBER01 also being empty in model_configure:

2.246 + 1> model_configure 0<< \EOF
total_member:            1
print_esmf:              .false.
PE_MEMBER01:
start_year:              2019

The setting of NTASKS_FV3 happens in exglobal_efsoi_forecast.sh (just like in exglobal_forecast.sh)

NTASKS_FV3=${NTASKS_FV3:-$npe_fv3}

Following npe_fv3 backwards...it comes out of config.fcst via npe_fcst:

export npe_fv3=$npe_fcst

...with a value of 424:

npe_fv3=424

...but then I see it getting reset in config.efcsfsoi and an error (line numbers included):

1098 0.061 + echo '  + 1 * 40'
1099 (standard_in) 1: syntax error
1100 0.063 + npe_fv3=''
1101 0.063 + export npe_fv3
1102 0.063 + npe_efcs=''
1103 0.063 + export npe_efcs

/scratch1/NCEPDEV/stmp4/Andrew.Eichmann/hedgehog/logs/2019111706/gdasefcsfsoi01.log

This part of that config is where the error comes from:

 14 export npe_fv3=$npe_efcs
 15
 16 if [ $QUILTING = ".true." ]; then
 17     export npe_fv3=$(echo " $npe_fv3 + $WRITE_GROUP * $WRTTASK_PER_GROUP" | bc)
 18     export npe_efcs=$npe_fv3
 19 fi

The value of $npe_fv3 on line 17 is unknown (cause $npe_efcs on line 14 is unknown) when it enters that block and causes the error that empties npe_fv3. I believe you want $npe_efcsfsoi instead of $npe_efcs on line 14 in config.efcsfsoi since this job isn't efcs so $npe_efcs doesn't come out of config.resources, $npe_efcsfsoi does. If you look a few lines above in the log (1083) you'll see $npe_efcsfsoi being set in config.resources.

So change $npe_efcs to $npe_efcsfsoi in config.efcsfsoi and rewind the failed job. Let me know how it goes.

[AndrewEichmann-NOAA]

This worked! Revealing once again the problem with proliferating config files.

[AndrewEichmann-NOAA]

Encountered another issue with the regular epos failing, it turning out because ensemble members 11-20 had no forecasts. This was traced back to the problem, previously enountered, with using NMEM_ENS=20 with export NMEM_EFCSGRP=2. Only five tasks of two members each get run. Solved by setting export NMEM_EFCSGRP=4 in config.efcsfsoi. Need to rig some way not get bit by this.

[AndrewEichmann-NOAA]

eposfsoi failing early due to $npe_node_eposfsoi apparently not being defined, as discovered in env/HERA.env. Set to $npe_node_epos, as it probably doesn't need to be different. Will have to review all EFSOI-specific environment settings to see if redundant.

[AndrewEichmann-NOAA]

Test running new build (experiment ibex) as pushed to repository, flushed out some problems with config files which hadn't been changed from Liaofan's versions. Merged with existing ecen and efcs configs. Edited, committed, and pushed. Also creating a test scripts to clone, build and run overnight.

[KateFriedman-NOAA]

@AndrewEichmann-NOAA Please update issue, thanks!

[AndrewEichmann-NOAA]

Tweaks to config files worked, closing. Thank you for your troubleshooting!