Calibration and Validation Overview

Calibration set-up

The Osmose calibration set-up is done by using the osmose_calibration_setup function, which creates a new calibration and runs some tests.

The current types are:

• simple, which uses biomass and landings
• survey, which uses the Osmose survey outputs

Calibration initialisation

First, a calibration folder needs to be initialised. This is done by a first call to the osmose_calibration_setup function. For instance, the initialisation of a simple calibration is done as follows:

jar_file = osmose:::.get_default_osmose_jar()
calibration_path = osmose_calibration_setup(input=configFile, 
                                            osmose=jar_file, 
                                            type="simple", 
                                            name = "my_first_calibration")

Generated files

This function creates a new folder named after the name argument preceded by calibration_ (calibration_my_first_calibration in the above example).

This folder contains the following files:

• calibration_OMP.pbs, calibration_MPI.pbs and calibration_sequentiel.pbs are the PBS scripts that can be used to run the calibration on Datarmor
• osmose-peru-parguess.osm, osmose-peru-parmax.osm, osmose-peru-parmin.osm and osmose-peru-parphase.osm are the initial guess, maximum and minimum values of the calibrated parameters and their associated calibration phase [@oliveros2017].
• osmose-peru-calibration_settings.csv contains the calibration parameters associated with all the variables used in the calibration.
• The calibration script .calibration.R, which is used to run the calibration (see the calibration_XXX.pbs files)
• The script that is used to run the Osmose model during the calibration process, run_model.R.

Note that the parameters that are needed by Osmose and that are usually calibrated (larval mortalities for instance) will be automatically added in the calibration process. Therefore, they are added in the osm files containing the parguess, parmin and parmax parameters.

Note

The calibration script is hidden. Therefore, it should not be edited manually

Warning

TODO: Finish description of the generated files

Data templates

The osmose_calibration_setup function also creates a folder called data_templates in the calibration folder, which contains file templates for calibration data. These files are automatically generated depending on the Osmose configuration that is calibrated. These files will be generated based on:

• The first year of output (output.start.year Osmose parameter) and the output frequency (output.recordfrequency.ndt parameter)
• The number and names of the focal species
• Whether output surveys (surveys.enabled.sr1) are used or not.

In the osmose-peru configuration, there are 9 focal species:

species.name.sp0 = anchoveta
species.name.sp1 = merluza
species.name.sp2 = sardina
species.name.sp3 = jurel
species.name.sp4 = caballa
species.name.sp5 = mesopelagicos
species.name.sp6 = munida
species.name.sp7 = pota
species.name.sp8 = euphausidos

and three output surveys:

surveys.name.sr1 = acousticSurvey
surveys.name.sr2 = trawlSurvey
surveys.name.sr3 = acousticCrustacean

Therefore, the data_templates will contain the following files

peru_biomass.acousticCrustacean-index_month.csv
peru_biomass.acousticSurvey-index_month.csv
peru_biomass-index_month.csv
peru_biomass.trawlSurvey-index_month.csv
peru_yield_month.csv
catch_at_length/peru_catchatlength-anchoveta_month.csv
catch_at_length/peru_catchatlength-caballa_month.csv
catch_at_length/peru_catchatlength-euphausidos_month.csv
catch_at_length/peru_catchatlength-jurel_month.csv
catch_at_length/peru_catchatlength-merluza_year.csv
catch_at_length/peru_catchatlength-mesopelagicos_month.csv
catch_at_length/peru_catchatlength-munida_month.csv
catch_at_length/peru_catchatlength-pota_month.csv
catch_at_length/peru_catchatlength-sardina_month.csv

i.e.:

• 1 spatially integrated biomass file (peru_biomass-index_month.csv)
• 1 spatially integrated yield biomass file (peru_yield_month.csv)
• 1 biomass file for each survey (peru_biomass.SURVEY-index_month.csv, with SURVEY changed to each survey name)
• 1 catch-at-length file for each species (catch_at_length/peru_catchatlength-SPECIES_month.csv with SPECIES the species name)

The next step for the user is to fill the files in the data_templates folder with the calibration data that will be used. These files need to be filled without modification to their structure. You can either leave them in the data_templates folder, but it is highly advised to copy the data_templates folder into a different location.

Correction of the calibration settings

By default, the calibration_settings file will include all the possible variables in the calibration process, based on the output of the Osmose simulation. However, in some cases, Osmose outputs may be missing observational based counterparts.

In this case, the calibration_settings.csv file must be manually edited and the calibrate column corresponding to the missing data must be set to FALSE. And the corresponding data files that were automatically generated in the data_templates folder must be removed.

For instance, in the osmose-peru configuration, the catch-at-length data for the mesopelagicos, munida, pota and euphausidos species are missing. Therefore, the initial calibration settings for these variables (@tbl-calibration-settings-init) are corrected as show in @tbl-calibration-settings-corrected.

Calibration finalisation

When the observed data have been filled or deleted (in case of missing folder), the osmose_calibration_setup function must be called a second time, with the data_path argument that specifies where the real calibration data are stored.

In the osmose-peru case, the data_templates files has been copied and pasted in a real_data folder. In the latter, the catch-at-length files for the mesopelagicos, munida, pota and euphausidos species have been removed and real data has been filled. Therefore, the calibration finalisation can be achieved as follows:

calibration_path = osmose_calibration_setup(input=configFile, 
                                            osmose=jar_file,
                                            type="simple", 
                                            name = "data_templates",
                                            data_path = "real_data")

Update of the run_model.R function

At this stage, the user also has the possibility to modify the run_model.R function, which makes the connection between observed and simulated data.

The user is invited to add modification between the BEGIN USER ADDED and END USER ADDED flags which are located in the run_model.R file:

  ### BEGIN USER ADDED

  ### END USER ADDED

Caution

In most cases, the run_model.R script won't need any modification. Proceed with caution if you do so

Running the calibration

The calibration can finally by run by using the .calibrate.R script. Since most calibration will be performed on High Performance Calculation (HPC), which are most of the time Linux-based systems, the calibration will be launched as follows:

Rscript .calibration.R >>& calibration.log

Note

Datarmor users are invited to run the calibration by using one of the .pbs scripts generated by the osmose_calibration_setup function:

qsub calibration_OMP.pbs # Open MP parallelisation
qsub calibration_MPI.pbs # MPI parallelisation
qsub calibration_sequentiel.pbs # No parallelisation