# OpenEPS (Open Ensemble Prediction System)

usage : ./openEPS [Options]

 Options 
   -e, --exp     Select experiment template to run
   -m, --model   Select model to run
   -r, --restart Remove existing exp data folder and do a
                 clean run
   -s, --silent  Do not print stderr and stdout on konsole
   -dr,--dry-run Only create a job, do not submit it directly
                 into a batchjob queue
   -h, --help    This help


This is a versatile Ensemble Prediction System that provides the
framework for running ensembles of different applications. The 
scripts are available for running OpenIFS. There is still quite 
much to do, and the structure will likely be changed until 
everything is clean and understandable.

So, please be patient when using the software!


 --------------------------------------------------------------
  Getting started:
 --------------------------------------------------------------
1) Setup an environment file if your HPC/linux cluster/etc.
is not among the env.-files in the repository. The env.-file
contains hard coded paths to the model bin, necessary libraries
that the model requires, information about the computing 
environment configuration etc. Have a look at the existing
env-files, copy one that resemples your system configuration 
the most and setup all the necessary information.

2) Create a scripts folder for your experiments in examples/oifs/,
e.g. scripts-seasonal. Any script in this folder will be used
instead of the default scripts (found in scripts/). Note, use
identical names to those found in scripts/. As an example: you
have need to link more files (or files with different storage
structure) to OpenIFS as is done in the default scripts/link.bash.
In this case you could simply copy link.bash to scripts-seasonal/.
and do any modification necessary there. In order to overwrite
the default files you simply provide SCR_MOD="seasonal" variable
in the experiment configuration file (see e.g. exp.example2).

3) Try running the example experiments provided, 2 and 5 are
recommended:
./openEPS -e example2
./openEPS -e example5

4) Navigate to the experiment folder and debug if you encounter
problems. Most likely problem is that env-file is not setup
correctly.


 --------------------------------------------------------------
  Structure:
 --------------------------------------------------------------
All common utility scripts and programs can be found in bin/.
A few example setups are available in examples/. These provide
guidance on how to setup and utilize openEPS for running 
ensembles of
1) OpenIFS with T21 resolution
2) OpenIFS with TL159 using a dataset of initial state 
   perturbations
3) OpenIFS with TL159 using the same dataset for initial state 
   perturbations but artificially increased perturbation amplitude 
   (requires CDO) and a model uncertainty representation
4) OpenIFS with a parameter estimation algorithm (EPPES) and how 
   to launch ensemble with varying global closure paramter values
5) OpenIFS with TL159 using a dataset of initial state 
   perturbations and utilizing wave model in each forecast. 


 --------------------------------------------------------------
  Workflow: (items in [] are TBD)
 --------------------------------------------------------------
./openEPS
1) Read in configuration files that define both system and model
   specific resources, paths, needed scripts, etc.
   (examples/$MODEL/configs/exp.$NAME)
   (examples/$MODEL/configs/env.$HOST)

2) [Test all needed files etc. are present]
   (bin/util_test.bash)

3) Initialize experiment
   - Create experiment structure (create paths, copy scripts)
   - Calculate resource allocation, i.e. how much can be run
     in parallel
   - Adjust model namelist(s) accordingly
   - Create explicit makefiles for all dates
   - IF parameter estimation: Initialize parameter estimation
   - Modify patch job resource requirements
   - Adjust post processing scripts  
   (bin/util_init.bash)  

 4) Submit batchjob, or launch a new bash shell
    - Loop over datelist, and let date specific makefiles
      handle workflow within ensemble for each date (see below)


 --------------------------------------------------------------
  About make:
 --------------------------------------------------------------
Job scheduling within each given date is handled via explicitly
written makefiles, the workflow within each makefile may depend
on the application. For instance, in the OpenIFS examples the
makefile run logic is the following:

exp.example2
1) link initial states for each ens member, final adjustments to
   namelists
2) once initial states are available, run the model
3) once model output is available, do some simple post-processing
4) once all ensemble members have completed 1-3, run a common
   post-processing script
5) go to next date

exp.example4
1) link initial states for each ens member, final adjustments to
   namelists
2) once initial states and namelists are available, change
   selected closure parameter values via namelists
3) run the model
4) once model output is available, calculate a cost function
5) once all ensemble members have completed 1-4 execute an EPPES
   update that will update parameter distributions, also draw
   new parameter values for the next date
6) go to next date


 --------------------------------------------------------------
  ISSUES/TBD (18th of September 2020):
 --------------------------------------------------------------
 

Pirkka Ollinaho, pirkka.ollinaho@fmi.fi
18th of September 2020