Metric Computations Interface

[melopeo]

Currently metric computations are basically done around make_evaluations_predictions and the Evaluator as shown in this example:

    predictor = estimator.train(dataset.train)

    forecast_it, ts_it = make_evaluation_predictions(
        dataset.test, predictor=predictor, num_samples=100
    )

    agg_metrics, item_metrics = Evaluator()(
        ts_it, forecast_it, num_series=len(dataset.test)
    )

It would be useful to provide a way to compute the metrics without having to construct any predictor at all.
For instance: I have forecasts generated by GluonTS and they are in the typical json format, and I have my actuals, then how can I compute the metrics?

I first design idea would be to have a function to call that goes like this:
agg_metrics, item_metrics = compute_metrics(fcsts, actuals, freq)

To make this happen some of the related tasks would be:

1. Set forecasts to SampleForecast
2. Set actuals as a ListDataset iterator

[jaheba]

I think our current interface is a bit odd: We return aggregated and item metrics as a tuple. So something like this looks more modular:

item_metrics = get_metrics(...)
agg_metrics = item_metrics.aggregate()

I wonder, how do I as a user define which metrics are calculated?

[mbohlkeschneider]

I don't think the metrics can be user specified at this point.

But it would make sense to me to have the the evaluator return item_metrics which are then aggregated.

I think it would be even better to provide a groupby-style operation so we can actually aggregate by some some categorical feature and or/custom fields.

For example:

agg_metrics = item_metrics.aggregate(by=['store', 'item_type'])

This is actually a use-case many people will have a lot.

[melopeo]

I think the design proposal of @jaheba makes a lot of sense, as there is no need to couple both item and aggregated metrics.

About the question of how do I as a user define which metrics are calculated? it seems to be possible as described here: One can pass which metrics to compute, what statistic to take as forecast, and what aggregation function to take for the computation of aggregated functions. Directly citing the above mentioned function:

custom_eval_fn
    Option to include custom evaluation metrics. Expected input is
    a dictionary with keys specifying the name of the custom metric
    and the values are a list containing three elements.
    First, a callable which takes as input target and forecast and
    returns the evaluation metric.
    Second, a string specifying the aggregation metric across all
    time-series, f.e. "mean", "sum".
    Third, either "mean" or "median" to specify whether mean or median
    forecast should be passed to the custom evaluation function.
    E.g. {"RMSE": [rmse, "mean", "median"]}

On how one can pass a user define metric seems not clear at the moment, as mentioned by @mbohlkeschneider .

About a groupby-style operation as mentioned by @mbohlkeschneider:
I think this would be possible with the requirement that the corresponding information, for instance by=['store', 'item_type'] is available in a suitable field, like item_id or similar.

[jaheba]

I think what we can do is to separate the definition of the metric and how it is used.

Each metric consists of two steps:

• calculate a metric-specific loss on a value to value basis
• aggregate losses of previous step

For example, MSE consist of

• calculating a loss: np.square(target - forecast)
• aggregate loss by taking the mean

class MSE:
    def loss(self, target, forecast):
        return np.square(target - forecast)

    def aggregate(self, loss, axis=None):
        return np.mean(loss, axis=axis)

The nice thing about this approach is, that we can use the aggregate methods multiple times:

targets = [[...]]
forecasts = [[...]]

mse = MSE()
loss = mse.loss(targets, forecasts)
item_metric = mse.aggregate(loss, axis=1)
agg_metric = mse.aggregate(item_metric)

Also, we can calculate the metric for each time-step, instead of each time-series:

mse = MSE()
loss = mse.loss(targets, forecasts)
mse_by_time = mse.aggregate(loss, axis=0)

[jaheba]

To improve efficiency, we could cache results of computation.

As a trivial example, MSE requires the loss, which we could calculate once doing loss = target - forecast.

class Loss:
    def get(self, x):
        x.loss = x.target - x.forecast

class MSE:
    requires = [Loss()]
    def loss(self, x):
        return np.square(x.loss)
    ...

---

Another thing we might want to support is dependencies of metrics.

For example RMSE is just defined as np.sqrt(totals["MSE"]).

Ideally, we can do this by writing something like:

class RMSE:
    dependencies = [MSE()]

    def get(self, metrics):
        return np.sqrt(metrics.mse)

[jaheba]

Something which complicates this is that we have things like mean_wQuantileLoss, where we average over all available quantile-losses:

class WeightedQuantileLoss:
    quantile: float

    def dependencies(self):
        return QuantileLoss(quantile), AbsTargetSum()

    def get(self, metrics):
        ql = QuantileLoss(quantile)
        ats = AbsTargetSum()

        return metrics[ql.name] / ats.name



class MeanWeightedQuantileLoss:
    quantiles: list[float]

    def dependencies(self):
        return [
            WeightedQuantileLoss(quantile)
            for quantile in self.quantiles
        ]

    def get(self, metrics):
        quantiles = [
            WeightedQuantileLoss(quantile)
            for quantile in self.quantiles
        ]

        return np.mean(
            [
                metrics[quantile.name]
                for quantile in quantiles
            ]
        )