Beschreibung in deutscher Sprache siehe hier.
Website:: https://kitmetricslab.github.io/forecasthub/
Preprint: https://www.medrxiv.org/content/10.1101/2020.12.24.20248826v2
Old version of visualization incl. evaluation scores: https://jobrac.shinyapps.io/app_forecasts_de/
Study protocol:: https://osf.io/cy937/registrations
Reference: Bracher J, Wolffram D, Deuschel, J, Görgen, K, Ketterer, J, Gneiting, T, Schienle, M (2020): The German and Polish COVID-19 Forecast Hub. https://github.com/KITmetricslab/covid19-forecast-hub-de.
Web tool to visualize submission files: https://jobrac.shinyapps.io/app_check_submission/
Web tool to explore forecast evaluations (still in development): https://jobrac.shinyapps.io/app_evaluation/
Contact: forecasthub@econ.kit.edu
This repository assembles forecasts of cumulative and incident COVID-19 deaths and cases in Germany and Poland in a standardized format. The repository is run by members of the Chair of Econometrics and Statistics at Karlsruhe Institute of Technology and the Computational Statistics Group at Heidelberg Institute for Theoretical Studies, see below.
An interactive visualization and additional information on our project can be found on our website here.
We are running a pre-registered evaluation study covering the months of October through March to assess the performance of different forecasting methods. You can find the protocol here.
The effort parallels the US COVID-19 Forecast Hub run by the UMass-Amherst Influenza Forecasting Center of Excellence based at the Reich Lab. We are in close exchange with the Reich Lab team and follow the general structure and data format defined there, see this wiki entry for more details. We also re-use software provided by the ReichLab (see below).
If you are generating forecasts for COVID-19 cases, hospitalizations or deaths in Germany and would like to contribute to this repository do not hesitate to get in touch.
We collect 1 through 30 day and 1 through 4 week ahead forecasts of incident and cumulative deaths by reporting date in Germany and Poland (national level), the German states (Bundesländer) and Polish voivodeships, with a special focus on short horizons 1 and 2 week ahead. This wiki entry contains details on the definition of the targets. There is no obligation to submit forecasts for all suggested targets and it is up to teams to decide what they feel comfortable forecasting.
Our definition of targets parallels the principles outlined here for the US COVID-19 Forecast Hub.
Up to 14 December we treated the ECDC data available here and here in a processed form as our ground truth for the national level death forecasts. As of 19 December, we use data we process directly from Robert Koch Institute and the Polish Ministry of Health see below. These agree with the ECDC data up to 14 Dec.
We collect 1 through 4 week ahead and 1 through 30 day forecasts of incident and cumulative confirmed cases by reporting date in Germany and Poland (national level), German states (Bundesländer) and Polish voivodeships, see the wiki entry. The respective truth data from RKI and the Polish Ministry of Health can be found here and here.
The main contents of the repository are currently the following (see also this wiki page):
data-processed: forecasts in a standardized formatdata-truth: truth data from JHU and ECDC in a standardized formatdata-raw: the forecast files as provided by various teams on their respective websites- The interactive visualization, which has been implemented by embers of the Signale Team at RKI, is maintained in a separate repository.
Submission for actively contributing teams is based on pull requests. Our wiki contains a detailed guide to submission. Forecasts should be updated in a weekly rhythm. If possible, new forecast should be uploaded on Mondays. Upload until Tuesday, 3pm Berlin/Warsaw time is acceptable. Note that we also accept additional updates on other days of the week (not more than one per day), but will not include these in visualizations or ensembles (if no new forecast was provided on a Monday we will, however, use forecasts from the preceding Sunday, Saturday or Friday).
We moreover actively collect forecasts from a number of public repositories in accordance with the respective license terms and after having contacted the respective authors.
We strongly encourage teams to visually inspect their final forecasts prior to submission. We created a Shiny app to help you in this process.
We try to provide direct support to new teams to help overcome technical difficulties, do not hesitate to get in touch.
We store point and quantile forecasts in a long format, including information on forecast dates and location, see this wiki entry for details. This format is largely identical to the one outlined for the US Hub here.
The forecasts assembled in this repository have been created by various independent teams, most of which provided a license with their forecasts. These licenses can be found in the respective subfolders of data-processed. Parts of the processing, analysis and validation codes have been taken or adapted from the US COVID-19 forecast hub where they were provided under an MIT license. All codes contained in this repository are equally under the MIT license. If you want to re-use materials from this repository please get in touch with us.
Data on observed numbers of deaths and several other quantities are compiled here and come from the following sources:
- European Centre for Disease Prevention and Control This used to be our preferred source for national level counts, but ECDC has switched to weekly reporting intervals on 14 Dec 2020.
- Polish Ministry of Health. We pull these data from this Google Sheet run by Michal Rogalski. This is our preferred source for Polish voivodeship level counts. The data are coherent with the national level data from ECDC. These data are coherent with the ECDC data up to 14 Dec. To align with the ECDC time scale we have shifted them by one day, see here.
- Robert Koch Institut. Note that these data are subject to some processing steps (see here) and are in part based on manual data extraction performed by IHME. This is our preferred source for German Bundesland level counts. The data are coherent with the national level data from ECDC up to 14 Dec.
- Johns Hopkins University. These data are used by a number of teams generating forecasts. Currently (August 2020) the agreement with ECDC is good, but in the past there have been larger discrepancies.
- DIVI Intensivregister. These data are currently not yet used for forecasts, but we may extend our activities in this direction.
Details can be found in the respective README files in the subfolders of data-truth.
Currently we assemble forecasts from the following teams. Note that not all teams are using the same ground truth data. (used truth data source and forecast reuse license in brackets):
- epiforecasts.io / London School of Hygiene and Tropical Medicine (ECDC; MIT)
- Frankfurt Institute for Advanced Studies & Forschungszentrum Jülich (ECDC; no license specified)
- ICM / University of Warsaw (ECDC; to be specified)
- IHME (JHU; CC-AT-NC4.0)
- Imperial College (ECDC; Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License)
- Johannes Gutenberg University Mainz / University of Hamburg (ECDC; MIT)
- KIT (ECDC; MIT) These are two simple baseline models run by the Forecast Hub Team. Part of these forecasts were created retrospectively, but using only data available at the respective forecast date. The commit dates of all forecasts can be found here.
- Karlen working group (ECDC; to be specified)
- KITCOVIDhub The
mean_ensembleandmedian_ensembleare two different aggregations of all submitted and eligible forecasts, see here. While the median ensemble is our pre-specified main ensemble, we also monitor performance of the mean ensemble. - LANL (JHU; custom)
- MIM / University of Warsaw (ECDC; to be specified)
- MIT Covid Analytics (JHU; Apache 2.0)
- MOCOS Group (ECDC; to be specified)
- ITTW (Universities of Ilmenau, Trier, Wroclaw, Warsaw) (ECDC; no license specified)
- itwm (Fraunhofer Institute for Industrial Mathematics ITWM)
- Institute of Global Health, University of Geneva / Swiss Data Science Center (ECDC; none given)
- University of Leipzig IMISE/GenStat (ECDC; MIT)
- UCLA Statistical Machine Learning Lab (JHU; cc-by-4.0)
- University of Southern California Data Science Lab(JHU; MIT) (MIT)
- YYG (JHU; MIT)
One of the goals of this forecast hub is to combine the available forecasts into an ensemble prediction, see here for a description of the current unweighted ensemble approach. Note that we only started generating ensemble forecasts each week on 17 August 2020. Ensemble forecasts from earlier weeks have been generated retrospectively to assess performance. As the ensemble is only a simple average of other models this should not affect the behaviour of the ensemble forecasts. The commit dates of all forecasts can be found here. Starting from 2020-09-21, our main ensemble is the median rather than the mean ensemble, as the former showed better performance in evaluations.
At a later stage we intend to generate more data-driven ensembles, which requires evaluating different forecasts, both those submitted by teams and those generated using different ensembling techniques. We want to emphasize, however, that this is not a competition, but a collaborative effort. The forecast evaluation method which will be applied is described in this preprint.
The following persons have contributed to this repository, either by assembling forecasts or by conceptual work in the background (in alphabetical order):
- Johannes Bracher
- Jannik Deuschel
- Tilmann Gneiting
- Konstantin Görgen
- Jakob Ketterer
- Melanie Schienle
- Daniel Wolffram
- US COVID-19 Forecast Hub run by the Reich Lab.
- Code repository for the SARS-CoV2 modelling initiative
Members of our group have contributed to the following papers and preprints on collaborative COVID-19 forecasting:
- J. Bracher, D. Wolffram, J. Deuschel, K. Goergen, J.Ketterer, A. Ullrich et al (2021): Short-term forecasting of COVID-19 in Germany and Poland during the second wave – a preregistered study
- J. Bracher, E.L. Ray, T. Gneiting, N.G. Reich: Evaluating epidemic forecasts in an interval format.
- L.C. Brooks, E.L. Ray, J. Bien, J. Bracher, A. Rumack, R.J. Tibshirani, N.G. Reich: Comparing ensemble approaches for short-term probabilistic COVID-19 forecasts in the U.S.
- E.L. Ray, N. Wattanachit, J. Niemi et al: Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S..
The Forecast Hub project is part of the SIMCARD Information& Data Science Pilot Project funded by the Helmholtz Association. We moreover wish to acknowledge the Alexander von Humboldt Foundation whose support facilitated early interactions and collaboration with the Reich Lab and the US COVID-19 Forecast Hub.
The content of this site is solely the responsibility of the authors and does not necessarily represent the official views of KIT, HITS, the Humboldt Foundation or the Helmholtz Association.