Allow multiple forecasters to share data #219
Description
From @ryantibs (slightly paraphrased):
Most interesting forecast analysis involve comparisons between forecasters. Right now get_predictions() and evaluate_predictions() each only operate on a single forecaster at a time.
Quick fix:
The way it's implemented now, we could easily extend get_predictions() or evaluate_predictions() so that could take multiple forecasters (would just be a simple wrapper over the current implementations). But this would be inefficient, because if I had a lot of forecast dates, there would either be a ton of duplicated data fetching. Assuming we're always considering the same forecast dates across forecasters (the most common case I believe) ... for get_predictions(), there would be duplication if my forecasters used highly overlapping data sets for training, and for evaluate_predictions(), there would always be duplication.
Desired solution:
A better solution would be to reimplement a multi-forecaster version of get_predictions() that looks something like this:
- for each common forecast date,
- fetch all the data needed by all the forecasters for that forecast date (here is where we could reduce a lot of duplication),
- train all the forecasters and store predictions.
You could still use caching in this setup, it would only improve things, but it's the order of operations that would help more. And similarly, we could reimplement a multi-forecaster version of evaluate_predictions() that looks something like this:
- for each common forecast date,
- fetch the response data needed for all the evaluations,
- then evaluate the predictions and store the results.