Clone this wiki locally
Welcome to the OptClim2 wiki! This is a draft being written in June 2022 for the archer2 branch.
OptClim is a software framework to that uses a cyclic workflow optimise parameters in models by generating configurations (parameter sets) of models, running them and then comparing model results against observations.
Models that include parametrised processes are commonplace. Among examples in atmospheric modelling are those representing cloud processes – their formation and processes concerning both rain and reflection/radiation. Parameters have ranges of possible values, and with OptClim these parameters can be tuned so models better represent past observations, and so can be projected forward with more confidence – e.g. to understand the impacts of the increasing CO2 concentrations on climate.
OptClim runs require:
- the definition of parameters, each with range of allowed values and default values (4 or 5 is typical)
- the location of each parameter in the namelists of the model, to permit it to be edited. Currently this information has to be added to the OptClim code specific to the model. (It should be table-driven in future) There is also the concept of a "metaparameter" whereby one user-defined parameter can cause an array, or multiple other dependent namelist parameters to be set.
- observations (single numbers, such as global averaged outgoing shortwave radiation, for a climate model) Typically one more than the number of parameters being optimised.
- user-provided code to generate simulated observations from the model results
- selection of a supported optimisation method, currently DFOLS is generally used in preference to the supported alternative of Gauss-Newton. This sets parameter values for the runs to be orchestrated by OpClim.
These are all held in a JSON file used by the OptClim software. A study - the workflow and model instances set up by OptCim - is generated in the directory holding the JSON file. The JSON file includes:
|Name||Name of the study (directory to be created in the directory of the JSON file)|
|baseRunID||This is a _prefix for the directory of a run, e.g. if yd, then runs are in yd001, yd002...|
|runCode||the Archer2 budget code agains which jobs are accounted|
|machineName||Must be "slurm" for Archer2|
|modelName||one of MITgcm, CESM, UKESM|
|study.referenceModelDirectory||the model directory that is cloned (Note - not used for UKESM)|
|optimise.dfols||settings for the dfols optimiser|
|Parameters||define each parameter: range of values, initial value to be used by OptClim)|
|postProcess||specify the code and added data used for generating simobs from model outputs|
|targets||the target values of the simulated observations|
|simulatedObservations||names and associated data for the simulated observations, to connect the outputs of the postProcess script to the optimiser|
The OptClim2 software provides the following:
- Run the optimiser script that assimilates all results to date and determines the next set of models to be run, with their corresponding parameter values.
- Queue an array of sequential jobs, one task for each of the models to be run. These are in state “held”
- Queue a job to await completion of all the array tasks, to run the optimiser script again.
- Clone each model and modify their parameters
- Start each model, the model scripts including a “release command” for its array task, this being run on completion of the model. For each cloning, a “base model” is replicated – one already tested and amended so it interfaces with OptClim as described below.
A glossary of terms used is in a separate wiki page.
The initial prototype of OptClim, termed OptClim1 was developed and used on the University of Edinburgh cluster Eddie in a collaboration between Prof. Simon Tett, Prof. Coralia Cartis and Dr Mike Mineter..... links to papers (Any other names to include, Simon?)
OptClim2 was coded by Prof. Tett. This added functionality and reimplemented some of the bash scripts of OptClim1 in Python.
The Archer2 branch of GitHub was developed with support from eCSE to port OptClim2 to Archer2 with minimum amendments to code and with extensions for the MITgcm, CESM2 and UKESM models.
For guidance on installation see https://github.com/optclim/ModelOptimisation/wiki/Installing-OptClim
A separate wiki page exists for each supported model. An example study for each exists on Archer2.
It is expected that first-time users will require some support and guidance - to help finalise their plans as well as with initial configuration. This is on a best-efforts basis, and can be requested via a list called optclim_developers in the email domain mlist.is.ed.ac.uk.
|runAlgorithm.py||added import of each new model's class - it imports all valid models; no other change|
|UKESM.py||Class for UKESM; child of ModelOptimisation; saves parameters to the stub directory zd001 etc.|
|MITgcm.py||Class for models using MITgcm|
|CESM.py||CLass for CESM2|
|config.py||Specific functions for SLURM|