Epidemic Model for COVIDAnalytics Research Effort
The repository contains code for a epidemiological model utilized in the research effort of our group called COVIDAnalytics, with projections on the website:
This repository contains multiple models, most of them being archived in the
/archive folder, but also the main and current DELPHI model (as of November 20th 2020) which we call V4.0. Below are the characteristics of each models:
- DELPHI V4.0 - The current version of code on the website. It is a modification of the implementation from V3.0 although the underlying model is the same.
- DELPHI V3.0 - This is the previous version of code. It is an improvement upon the version of the model, as it takes into account interventions being lifted for a little while, causing a resurgence in the number of cases/deaths. This resurgence is modeled using a 3 parameters normal distribution added to our gamma(t) function (cf.
- Adaptive Policy Model / Continuous Retraining - This is an experimental model we tried implementing that would continuously retrain the parameters associated to the lifting or implementation of a policy in a given area and was supposed to help model a resurgence in cases. It unfortunately didn't yield good enough results and we decided to discard it.
- V1 - No Jump: Initial DELPHI Model used until late May 2020.
- V2 - Discrete Jump: This is an attempt at modeling a resurgence in cases using a simple discrete jump (similar to what is done with policy evaluations). This implementation was unsuccessful.
- V3 - Normal Jump + Trust Solver: Very similar implementation to the final V3.0 version we are currently using (as of September 5th 2020) where we use a trust region constrained solver instead of a TNC solver. The newer implementation actually allows for either so this can be discarded.
- V4 - Arctan Jump: This is an attempt at modeling a resurgence in cases using an arctan jump. This implementation was unsuccessful.
- Other analyses: contains a bunch of other analyses and modeling experiments we have conducted, among which other second wave modeling (as opposed to resurgence). They were unsuccessful.
We provide two implementations for the previous V2.0: The (deprecated) Mathematica version
archive/V1 - No Jump/COVID-19_MIT_ORC_training_script_global_V2.nb,
and the python3 version
archive/V1 - No Jump/DELPHI_model.py. The Mathematica notebook was written with Mathematica
12.1 but should work with any version greater than 10.0. The Python3 version is tested with Python 3.7.
The currently used implementation on the website and all other extra analyses for DELPHI V4.0 is in python3 under file
DELPHI_model_V4.py and is currently the final version. The policy-based predictions utilize the file
DELPHI_model_V4_with_policies.py which uses the outputs from
The latest documentation for the model is contained in the pdf document:
Please Cite the following when you are utilizing our results:
ML Li, H Tazi Bouardi, O Skali Lami, N Trichakis, T Trikalinos, D Bertsimas. Forecasting COVID-19 and Analyzing the Effect of Government Interventions. (2020) submitted for publication.
V4.0 How To Run Instructions
numpy, scipy (>=1.4.1), pandas, us
To run the V4.0 model successfully, you would require the following files for each region:
- Historical Case Files - This should be provided in the same format as the examples given in folder
data_sandbox/processed. The location of the files should be at
- Population File - This file should record the population at each location that needs to be predicted.
An example of such is in
data_sandbox/processed/Global/Population_Global.csv. The location of this file should be at
- Historical Parameter Files (optional) - This file record previously trained parameters and the optimization bounds would be within 10% of the original trained parameters. This should be provided in the format given in the example file
data_sandbox/predicted/Parameters_Global_20200621.csv. The location of the files should be at
- A run config YAML file at
./run_configs. This file is used to pass the settings for a particular run, some examples are given in the
./run_configsfolder and can be modified according to the need. It is discussed in more detail below.
File Paths for Python
To run the model successfully for python, please first add a new user in the
config.yml file and record the appropriate absolute paths:
delphi_repo: This is the local location for this repo.
data_sandbox: This is the location for policy data used in DELPHI V3.0 (only necessary for DELPHI V3.0).
danger_map: This is the location for saving the final predictions and loading the case files.
website: This is only utilized internally for publishing on the website, and could be ignored.
logs: This is the local file path to the logs folder inside the DELPHI repository.
Command Line Interface and Run Config
In order to run the model, run the following command on your terminal:
python3 DELPHI_model_V4.py --run_config <path to run-config.yml>
or a shorter version of it:
python3 DELPHI_model_V4.py -rc <path to run-config.yml>
If one wants to run the policy model, the following command should be run on the terminal:
python3 DELPHI_model_V4_with_policies.py --run_config <path to run-config.yml> or a shorter
version of it:
python3 DELPHI_model_V4_with_policies.py -rc <path to run-config.yml>.
The run-config file should have the following information:
userrunning the code, with its file paths referenced in the
config.ymlfile, otherwise the script will throw an error.
optimizermust be one of the three currently supported in our implementation (
annealing), otherwise it will throw an error. It is also important for the policy predictions in order to know from which optimizer the parameters that will be used will come from.
confidence_intervalsparameter must be a 0 (for False) or 1 (for True), depending on whether or not the user wants a final output containing confidence intervals on the number of cases and deaths (like the ones generated for the website). We advise users of this codebase to use 0 as default.
since100caseallows to save (or not) a prediction file starting from the date at which each area had its 100th case (varies from one area to another) on top of the file for which predictions start on the day of running the script. This is especially useful when one wants to evaluate model fitting on historical data.
- Finally, the
websiteparameter allows to choose whether or not to save the prediction and parameters files on the
DELPHI/websiterepository (default should be 0).
Backtest How To Run Instructions
Very similarly, to perform a backtest of the model (computing certain metrics on number of cases and number of deaths) one should just use the Command Line Interface running the following command:
python3 DELPHI_backtest.py --user <USER_RUNNING> --prediction_date <YYYY-MM-DD> --n_days <INTEGER> --mse <0 or 1> --mae <0 or 1> or
python3 DELPHI_backtest.py -u <USER_RUNNING> -pd <YYYY-MM-DD> -n_days <INTEGER> -mse <0 or 1> -mae <0 or 1>
USER must have its file paths referenced in the
config.yml file, otherwise the script will throw an error.
prediction_date must have the correct format, otherwise the script will throw an error.
n_days needs to be an integer. The script will automatically check that the backtest is feasible given the available historical
and prediction data in the
danger_map folder and the two latter inputs from the user running the script.
mae must be 0 or 1, depending on whether or not the user wants to compute MSE and MAE as well. The default
metric is MAPE and is always computed for both cases and deaths.
Compare Performance of Annealing and TNC
DELPHI_compare_V4.py can be run to compare the performance of Annealing and TNC at province level using the Command Line Interface in the following way:
python3 DELPHI_compare_V4.py --user <USER_RUNNING> --run_model <RUN_MODEL> --plots <PLOT_OPTION>
python3 DELPHI_compare_V4.py -u <USER_RUNNING> -r <RUN_MODEL> -p <PLOT_OPTION>
As mentioned for other use cases,
USER should have file paths in the
config.yml file. If the
run_model option is 1, the script will run the model with TNC and Annealing consecutively and then compare the metrics, otherwise predictions till current day with annealing and tnc both should be present in the
covid19orc/danger_map/predicted and the script will automatically read those. If
plot_option is 1, it will save the plot comparing predictions for every region in the
data_sandbox folder. The metrics used for default are KL divergence and Maximum Absolute Percentage Error.