Bootstrap wrapper for metrics?

[maximsch2]

🚀 Feature

We should provide ability to compute bootstrapped confidence intervals for metrics.

Motivation

Confidence intervals are important and we should make it easy for people to increase rigor of their research and model evaluations.

Pitch

I'm thinking we can have something like this (very high level):

class Bootstrapper(Metric):
   def __init__(self, num_samples, metric):
       self.metrics = nn.ModuleList([deepcopy(metric) for _ in range(num_samples)])

  def update(self, preds, targets):
     for idx in range(self.num_samples):
        preds_sampled, targets_sampled = sample_for_bootstrap(preds, targets)
        self.metrics[i].update(preds_sampled, targets_sampled)

which will let people to wrap any metric, have a set of copies of the metric internally updated with different samples of the data, giving us then ability to get a distribution of metric values.

Alternatives

We can skip it on the class-based metrics side and assume anyone doing bootstrap will load everything in memory and do bootstrap using functional metrics.

[SkafteNicki]

Hi @maximsch2, great idea. Would you be up for doing a PR?
Just to be clear, would the compute method look something like:

def compute(self):
    computed_vals = torch.stack([m.compute() for m in self.metrics], dim=0)
    return computed_vals.mean(dim=0), computed_vals.std(dim=0)

or would you just return the computed values?

[maximsch2]

Yes, we need to decide how we want to represent the bootstrapped result (e.g. mean + std, or maybe 5%, 50%, 95% percentiles, possibly even configurable), but yes, your compute function is approximately what I have in mind.

I don't have too much bandwidth to do it right now, so throwing it out there if someone wants to take it on. I might get back to it in the future though.