Make predictions in a new or held out dataset

[raamana]

Ability to input a new dataset, from a different site or dataset or country, and use the best model to report performance on this dataset

Or an option to specify attribute-based criterion to hold a certain subset out completely to report performance

[raamana]

An obvious issue to be solved is the definition of what the best model is — one parameter combination is only evaluated once, and a simple numerical comparison of accuracy isn’t a good/robust way pick it.

Best model could be defined by the Param combination that was most frequently selected over N>100 reps of the inner CV loop (I report it for user FYI), but often there are many within the same freq range of 30-40%, and we could employ some non-parametric stats there to pick one!

CLI option could be —report_on