Meta-model Runtime Components

[rburghol]

Tasks

• Devise function plugin architecture/ Naming convention options:
  • Functional directories, supporting name order multiple scripts
  • Examples:
    • models/[model name]/river/prep/
    • models/[model name]/river/run/
    • ...
• Confirm design/direction with Gopal
• Basic testing with slurm Run with slurm #3
• Advanced slurm integration Run with slurm #3
• Use . plugins/[model]/model_config to load env vars
• review/check in with NOAA next GEN model. Do they already have a meta-model run time frame work?
• Plugin Directory Structure details #4
• How to handle object class specific summary scripts (see "object class summary scripts "below)
• Create shared function to require a specific list of environment variables.
• Document data Transfer, execution steps:
  • list of WDM DSNs
  • model run metrics, and OM functions (need to separate the log file retrieval from the analysis in the old OM functions)

Plug-in function design

• run_model
  • Use: run_model model scenario segment run_directory [modules all/land, river] [submodule all/prep,run] [step all/01,02, or full name "01_run_rug" or function name "run_rug"]
  • Ex: run_model cbp_hsp2 vadeq_2021 JL3_xxxx_yyyy all
    • run all modules, sub-modules and steps
  • Ex 2: run_model cbp_hsp2 vadeq_2021 N51003 land all
    • only want to run land lug, run, postproc, analyze,
  • Command switches:
    • -use-slurm: 1/0
    • -slurm-jobname
    • -slurm-dependency
    • -slurm-output
• Functional directories, supporting name order multiple scripts:
  • Model / Module / submodule / script(s)
    • Model: cbp6_hspf, dyn_model, cbp_hsp2
    • Module: land, river, ...
    • Submodule: prep, run, link, analyze
    • scripts: 01_run_lug, 02_hsp_version
• Plugin directory structure (details here: )
  • models/[model name]/[module]/
    • models/[model name]/[module]/prep/
    • models/[model name]/[module]/run/
    • models/[model name]/[module]/post/
    • models/[model name]/[module]/analyze/
• Ex hsp2:
  • models/hsp2/river/prep/01_run_rug (A soft link to run/standard/run_lug.csh)
  • models/hsp2/river/prep/02_convert_uci (A short script to call the HSPF to hsp2 conversion routine)
  • models/hsp2/river/postproc/01_export_to_wdm used in stream_wdm.exe

Object class summary scripts

• summary scripts for facility objects, as well as for the case when I UCI has multiple river segments and land uses, will require looping through all the possibilities.
• This is somewhat different from OM simulation which explicitly calls a separate summary script for each element.
• We can opt to have a sub folder to have different object class specific ones and have a function that loops through calling each, but we have to ensure that we check the file when we are looping through the main loop to see if it's a folder or not and if it's a folder skip it.

other function naming conventions considered

1. Directory based with identical plugin names, ie.:
   • ./models/[model name]/export_land
   • ./models/[model name]/summary_river
   • ./models/[model name]/etm
   • Ex: ./models/hsp2/summary_river
2. Functional description for the Scripts that do CBP model integration steps, exports and data alias translations, and which call summary script. These should be defined with the naming convention to indicate function as well as model domain, i.e.,
   - cbp_etm_hsp2 (currently not used)
   - cbp_land_export_hsp2: will do the post-processing of land export files
   - cbp_river_export_hsp2: will do the post-processing of river export files
   - cbp_river_summary_hsp2: will do the summary stats/REST setting of export files
   - cbp_river_stream_hsp2: will perform the translation of hydro to w

[rburghol]

#/bin/bash
# load environment if not already loaded
if (! $?SCRIPT_DIR) then
  . hspf_config
  source $CBP_ROOT/config/control/script/${scenario}.con
endif
 
scenario=$1
segment=$2
# set up places to look for plugins
model_plugins=$CBP_ROOT/plugins/$model
module="river"
for steps in "prep run postproc analyze"; do
  if [ -d $model_plugins/$module/$steps ]; then
    for plugin in `ls $model_plugins/$module/$steps/`; do
      $model_plugins/$module/$steps/$plugin $scenario $segment
    done
  fi
done

[rburghol]

Use examples:

• only want to run land run, postproc, analyze,
  • must pass in model, modules.
• only want to run river prep, run, postproc, analyze

Arguments:

• run_model scenario segment model run_directory [modules all/land, river] [submodule all/prep,run] [step all/01,02, or full name "01_run_rug" or function name "run_rug"]

[rburghol]

This now works:

• runs all components in river
• currently uses hspf, but UCI is tweaked to allow hsp2
• Now need to add summary script test and communicate with harp

/opt/model/meta_model/run_model hsp2_cbp6 vadeq_2021 JL3_7090_7150 test river

[rburghol]

script to insert positional script.

model_bump step 

#!/bin/bash
inpos=$1
nextpos=$((inpos + 1))
# if there are no files in the next slot, we just move this slots files up one
# otherwise, we shuffle everything up 1 slot
i=98
while [ $i -ge $inpos ] ; do
  j=`printf "%02g" $i`
  echo "$j"; 
  movers=`ls ${j}_* 2>/dev/null`
  for onefile in $movers; do
    num=${onefile:0:2}
    newnum=$((10#$num + 1))
    newindex=`printf "%02g" $newnum`
    newfile=${onefile/$num/$newindex}
    echo "Moving: $onefile $newfile"
    mv $onefile $newfile
  done
  i=$((i - 1))
done

[rburghol]

Examples:

• run all in run phase, hsp2: /opt/model/meta_model/run_model hsp2_cbp6 hsp2_2022 JL1_7080_7190 h2_test river run
• run just the 8th step, 08_run_river in run phase, hsp2: /opt/model/meta_model/run_model hsp2_cbp6 hsp2_2022 JL1_7080_7190 h2_test river run 08_run_river
• run single step in run phase hspf: /opt/model/meta_model/run_model hspf_cbp6 hsp2_2022 JL1_7080_7190 h2_test river run

[rburghol]

Run only the organic scour routine (not needed for hydro only, but if there are errors, this can be examined):

/opt/model/meta_model/run_model hsp2_cbp6 hsp2_2022 JL1_7080_7190 h2_test river run 11_scour_org

[rburghol]

• Ran: /opt/model/meta_model/run_model hsp2_cbp6 hsp2_2022 JL1_7080_7190 /opt/model/p6/vadeq/tmp/rob-scratch/h2_test river run 11_scour_org
• Error:

ERROR:  the wdm file
../../../tmp/wdm/river/hsp2_2022/stream/JL1_7080_7190.wdm
 Had zero flow for the first month

• but the wdm DOES have flow in the first month. Maybe the problem file from the previous still exists?
• Try deleting it. Boom!!! It works!
• Note: this is a bug in the model controllers atomized way of doing things problem files must be eliminated at the beginning of each step.