Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scenario comparison for deterministic models #225

Merged
merged 6 commits into from
May 7, 2024

Conversation

pratikunterwegs
Copy link
Collaborator

This PR is WIP #221 and adds basic intervention scenario comparison functionality. The function outcomes_averted() offers options for calculating the differences in outcomes averted (cases, and deaths where applicable) between a baseline scenario (with parameter uncertainty), and multiple comparator scenarios (with identical parameter uncertainty).

  • See the vignette on scenario modelling, modelling_scenarios.Rmd for how to use this.
  • There's a limitation that this only works for models with parameter uncertainty. This is because the output format of a model with a single parameter set does not preserve model arguments/infection parameters which are used for matching. This in turn is because it was felt that users should be able to run a non-vectorised model and get simplified output that can be passed directly to plotting (see Vector inputs for ODE models #176).
  • There's no support for comparing scenarios where the axis of variation is an infection parameter, e.g. $R_0$ is one of 1.3, 2.0, 2.5 etc.
  • Support for the Ebola model is forthcoming in a separate PR based on feedback from this one.

Happy to take feedback on the implementation here, and more general discussion on scenario comparison in #221.

@pratikunterwegs pratikunterwegs self-assigned this Apr 30, 2024
@pratikunterwegs pratikunterwegs added the Scenarios Related to scenario comparison functionality label Apr 30, 2024
Copy link

This is how benchmark results would change (along with a 95% confidence interval in relative change) if a6e001e is merged into main:

  • 🚀default_ode: 10.6ms -> 10.5ms [-1.79%, -0.01%]
  • ✔️default_ode_interventions: 97.2ms -> 95.9ms [-2.98%, +0.47%]
  • ✔️default_ode_param_vec: 845ms -> 842ms [-1.4%, +0.76%]
  • ✔️default_ode_paramvec_intervs: 6.19s -> 6.17s [-1.27%, +0.7%]
  • ✔️ebola: 577ms -> 575ms [-1.44%, +0.83%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Copy link

This is how benchmark results would change (along with a 95% confidence interval in relative change) if a6e001e is merged into main:

  • ✔️default_ode: 10.7ms -> 10.6ms [-2.01%, +0.96%]
  • ❗🐌default_ode_interventions: 93.4ms -> 94.6ms [+0.02%, +2.4%]
  • ✔️default_ode_param_vec: 819ms -> 819ms [-0.94%, +0.92%]
  • ✔️default_ode_paramvec_intervs: 6.2s -> 6.2s [-0.65%, +0.42%]
  • ✔️ebola: 550ms -> 549ms [-1.44%, +1.23%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Clean up rate intervention title

Rm extra baseline
Copy link

This is how benchmark results would change (along with a 95% confidence interval in relative change) if a6e001e is merged into main:

  • ✔️default_ode: 10.8ms -> 10.8ms [-0.48%, +1.07%]
  • ✔️default_ode_interventions: 97.7ms -> 97.7ms [-1.02%, +1.04%]
  • ❗🐌default_ode_param_vec: 832ms -> 841ms [+0.14%, +1.9%]
  • ✔️default_ode_paramvec_intervs: 6.24s -> 6.24s [-0.62%, +0.54%]
  • ✔️ebola: 563ms -> 562ms [-1.07%, +0.56%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@pratikunterwegs pratikunterwegs marked this pull request as ready for review April 30, 2024 14:18
Copy link
Member

@joshwlambert joshwlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pratikunterwegs nice work. All looks good to me. The new documentation in the vignettes makes the new functionality very clear. Left a couple of minor comments in the code.

R/scenario_comparison.R Outdated Show resolved Hide resolved

```{r}
# run each scenario for two values of R
# no parameter uncertainty
r_values <- c(1.3, round(r_estimate_mean, 2))
r_values <- c(1.3, 1.6)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving a comment here for figure 2 below. The x-axis labels are overlapping on the pkgdown site, making it hard to match the column with the intervention.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep - unsure how to fix this. I tried setting n.dodge = 3 but this still has overlaps.

Copy link

github-actions bot commented May 7, 2024

This is how benchmark results would change (along with a 95% confidence interval in relative change) if a6e001e is merged into main:

  • ✔️default_ode: 10.8ms -> 10.8ms [-0.83%, +1.14%]
  • ✔️default_ode_interventions: 100ms -> 100ms [-1%, +1.42%]
  • ✔️default_ode_param_vec: 837ms -> 835ms [-0.68%, +0.39%]
  • ✔️default_ode_paramvec_intervs: 6.24s -> 6.26s [-0.1%, +0.79%]
  • ✔️ebola: 570ms -> 566ms [-1.57%, +0.14%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@pratikunterwegs
Copy link
Collaborator Author

Thanks @joshwlambert. I haven't found a good fix for the second issue, but I'm merging this now so I can move ahead with more scenario functionality.

@pratikunterwegs pratikunterwegs merged commit 42b87b0 into main May 7, 2024
11 checks passed
@pratikunterwegs pratikunterwegs deleted the feature-scenario-compare branch May 7, 2024 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scenarios Related to scenario comparison functionality
Development

Successfully merging this pull request may close these issues.

None yet

2 participants