Calibrating a Model

Who this is for: the person who calibrates a project's model, usually one lab member per project. Before you start: a project built from the EpiModelHIV-Template that runs through chapter B (Getting Started with EpiModelHIV), network estimates on the HPC from the chapter A workflow, and Running Project Workflows on the HPC.

Calibration adjusts uncertain model parameters until the model's outputs match observed targets. This page explains how we calibrate EpiModelHIV models with the Template's chapter C, R/C-calibration/, and what goes wrong in practice. The examples and lessons come from the Template and from the calibration of our LA-PrEP CABARET project. The chapter's README lists every file.

The pieces

• Targets. EpiModelHIV::get_calibration_targets() returns the default targets, such as PrEP coverage (cc.prep.B), the share of people with HIV who are diagnosed (cc.dx.B), viral suppression (cc.vsupp.B), STI incidence (ir100.gono), HIV prevalence (i.prev.dx.B), and population size (num). EpiModelHIV::mutate_calibration_targets() computes the same quantities from simulation output. The race-specific targets end in .B, .H, and .W.
• Calibrated parameters. The parameters we adjust are the uncertain ones: PrEP start rates, HIV testing rates, treatment stop rates, HIV and STI transmission scalers, AIDS mortality, and the arrival rate. Their values live in the project's parameter overrides, data/input/model_parameters.csv (see EpiModelHIV ‐ New Parameters and Attributes).
• Assessment. process_calibs.R writes calib_assess.csv, the mean deviation of each target in each scenario, and 2-manual_calib_assess.R plots the target trajectories.
• Restart point. Calibration runs after the first phase start from a saved burn-in instead of from scratch. See Making a Restart Point.

If your project uses targets other than the defaults, define them once (for example in a project_calibration_targets() function in R/calibration_targets.R) and use that function everywhere targets are read: swfcalib_config.R, 3-choose_restart.R, and process_calibs.R. In one of our projects the new targets existed only in an issue while every script still read the old defaults, so a calibration wave would have fit the wrong numbers without any error.

The four phases

1. Ballpark (manual). Run the model from scratch over a grid of parameter values until every epidemic is present and every target is in the right order of magnitude.
2. First restart point. Save a simulation from the best ballpark scenario as the starting point for the next phases.
3. Fine-tuning. Adjust by hand from the restart point if needed, then let swfcalib converge on the targets.
4. Final restart point. Run the calibrated model from scratch many times and save the best simulation as the restart point that the intervention runs use.

Phases 2 and 4 are on Making a Restart Point.

Phase 1: ballpark calibration

workflow-ballpark_calib.R builds a workflow that runs a set of scenarios from step 1 to calibration_end (70 years in the Template), merges the output, and writes data/run/calibration/calib_assess.csv. Edit the scenarios_df tibble in the script to set the grid, for example:

n_scenarios <- 10
scenarios_df <- tibble(
  .scenario.id = as.character(seq_len(n_scenarios)),
  .at = 1,
  gono.uret.prob = seq(0.20, 0.22, length.out = n_scenarios),
  chla.uret.prob = seq(0.20, 0.22, length.out = n_scenarios),
  syph.prob = seq(0.115, 0.15, length.out = n_scenarios)
)

Deploy it as on Running Project Workflows on the HPC, download calib_assess.csv when the END email arrives, and look at the trajectories with 2-manual_calib_assess.R. Move the values in your parameter overrides toward the best scenario and repeat until the model is roughly right. 1-ballpark_calib.R does the same at small scale on your computer.

Phase 3a: manual fine-tuning

workflow-restart_calib.R is the same loop, but every simulation starts from the restart point (step 2) instead of from scratch. Runs take about as long, but the results are less noisy because the population was produced by the model rather than built from the network estimates. When you find better values, put them in your overrides and make a new restart point before starting swfcalib.

Phase 3b: automated calibration with swfcalib

swfcalib runs the calibration as a slurmworkflow that loops until every target is met. swfcalib_config.R defines it:

• Waves run one after another. Each wave holds jobs that run in parallel, and each job fits one or more parameters to one or more targets.
• Each job starts from initial proposals, values spread evenly across a prior range, and uses a proposer to make the next set: a shrink proposer, which halves the range around the best values each iteration, for the smooth care cascade targets, and an SE-range proposer, which keeps the best 30%, for the noisier STI and HIV prevalence targets.
• Each job has a convergence rule. When every job in a wave has converged, its values are fixed and the next wave starts from them. Parameters not yet calibrated take their values from default_proposal, which comes from your parameter overrides.

The Template's waves encode the order in which the model's parts depend on one another: the care cascade first, then transmission.

Wave	Targets	Parameters
1	cc.prep.B, cc.prep.H, cc.prep.W	prep.start.rate_1 to _3
2	cc.dx.B, cc.dx.H, cc.dx.W	hiv.test.rate_1 to _3
3	cc.vsupp.B, cc.vsupp.H, cc.vsupp.W	tx.halt.rate_1 to _3
4	ir100.gono, ir100.chla, ir100.syph	gono.uret.prob, chla.uret.prob, syph.prob
5	i.prev.dx.B, i.prev.dx.H, i.prev.dx.W	hiv.trans.scale_1 to _3
6	disease.mr100	aids.off.tx.mort.rate
7	num	a.rate

workflow-swfcalib.R builds the campaign. Each iteration runs 256 simulations, 8 per array task, from the restart point. When all waves have converged, the workflow writes the calibrated values, runs 256 replications with them, merges them, and writes calib_assess.csv. A full campaign can take a day or more: in the LA-PrEP CABARET project, a rerun of four waves at 512 simulations per iteration was planned at about 20 hours from the previous campaign's timings.

Before you launch a campaign

Each of these has silently wasted a campaign in one of our projects:

• Check that each target can be reached inside its prior. swfcalib does not detect an unreachable target: it settles at the edge of the prior and spends its whole iteration budget there. Before a campaign, run a small sweep of the parameter across its prior range (a ballpark-style workflow with a grid of that parameter) and check that the mean curve over replications crosses the target. Use the mean, not the spread of individual replications, which can make an unreachable target look reachable.
• Archive the old swfcalib state. calibration_step1() reads data/run/swfcalib/calib_object.rds if it exists and ignores the configuration you just built. A finished old campaign there makes the new one look complete at once, write the old values back, and send an END email as if it had worked. On the HPC, move data/run/swfcalib/ aside before each new campaign.
• Make sure the restart point matches the model. Every calibration simulation starts from the restart point. If you have added attributes since it was made, rebuild it first (Making a Restart Point).
• Make sure your overrides hold current values for the waves you skip. To rerun only some waves, subset calib_object$waves in swfcalib_config.R (for example to waves 4 to 7). The skipped waves' parameters then come from your overrides through default_proposal, so stale values there carry into the result.
• Push the code and renv.lock.hpc before deploying, as for any workflow.

Watching a campaign

• squeue -u $USER -o "%i %j %T %R": the campaign's jobs are named swfcalib_step1, swfcalib_step2, and so on, plus swfcalib_controller.
• Each step 1 log in workflows/swfcalib/log/ prints the current wave and iteration and the values in default_proposal, which fill in as waves converge.
• data/run/swfcalib/ holds assessments.rds (how each job's fit improves over the iterations) and results.rds (every simulation's parameters and outputs). Copy them to your computer and use 4-swfcalib_assess.R, which renders an assessment report with swfcalib::render_assessment() and plots parameters against targets.
• To stop a campaign, cancel swfcalib_controller first, then the running step, by name (see Monitoring and Managing Slurm Jobs).

After it converges

1. swfcalib writes calibrated.csv in data/run/swfcalib/. The workflow's update_param.R step then writes params.csv there: your current overrides plus the calibrated values.
2. Copy params.csv to your computer, check the values against calibrated.csv, and save it as data/input/model_parameters.csv. Commit and push.
3. Make the final restart point with the calibrated values (Making a Restart Point).
4. Render a record of the calibration with 5-calibration_report.R.

When to recalibrate

Recalibrate after any change that moves the targets: new targets, a change to a module that affects the care cascade or transmission, or an engine update that changes the dynamics. You do not always need every wave. In the LA-PrEP CABARET project, PrEP coverage, diagnosis, and viral suppression moved less than 0.4 percentage points across the whole range of STI transmission that the STI wave explored, so after a change to the STI targets only waves 4 to 7 were rerun. Check this in your own model before skipping waves. When a model change also adds attributes, the order is: make a new restart point, recalibrate, put the calibrated values in your overrides, and make the final restart point again.

Next: Making a Restart Point.