Test model on all labels from the future

[jtwalsh0]

Triage should have an option to assess the performance of a model on any future test set, i.e. any test set that begins after the train set ends. For an annual-prediction example, a model trained on data ending December 31, 2009, should be tested on all test sets that begin January 1, 2010; January 2, 2011, January 3, 2012; and so on.

This would help us understand how often the model should be retrained and what the partner loses as models get older. It might also help identify problems or interesting patterns, where model performance degrades after a while but increases again.

The user should be able to set a parameter (what to call it, "max time difference"?) that would only test the model on labels within x time of the end of the train data. In the above example, the user might limit testing a model trained on 2009 data to test sets from 2010, 2011, and 2012 and not 2013 on.

[nanounanue]

This is supported if you use the correct temporal config , isn't ? (but it is a pain)

[ecsalomon]

I propose an alternative solution:

1. An additional key in the temporal_config for prediction_frequency. Triage in default mode will continue making test matrices and testing models every prediction_frequency until the model_update_frequency, at which point, it will train a new model and use that for predictions going forward until it hits the model_update_frequency again. For example if prediction_frequency is one month and model_update_frequency is three months, it will generate three sets of predictions from each model. Of course, we'll need to address Better handling of multiple prediction windows #327 to make this truly useful. This would allow projects to jumpstart with a long retraining window while still generating predictions as frequently as required by the context to narrow down the learner grid more quickly.
2. A predict-everything-after-the-model mode that is used strictly for deciding on retraining frequency. This would basically be like @jtwalsh0's suggestion, but per discussion with @Rayid is not something we want to do by default on our first experiments, as it is costly.

[ecsalomon]

I thinnnnk the experiment implementation of # 1 is fairly trivial. It should just be a timechop change, as timechop is already written to pass lists of test matrices out to the other components (it just only puts one item in the list), so if the other components are anticipating lists with loops and not just grabbing the first element, the tests would likely be hardest part.