[ENH] forecasting evaluate utility failing with quantile forecasts

[fkiraly]

There is a failure on main related to evaluate failing with quantile forecasts, specifically test_evaluate_with_window_splitters. Full diagnostic output can be seen here: https://github.com/sktime/sktime/actions/runs/6379899253/job/17313415803?pr=5083

As the full suite ran through with the 0.23.0 release, the regression most likely has been introduced with the only PR since then that changed the evaluate logic: #5192 (the test depends on probabilistic metrics and evaluate, and only one of the two have changed since 0.23.0)

The CI for 5192 did not detect this as we have conditional testing, and the failing test is not registered as to be triggered by changes in evaluate, unlike the tests in test_evaluate, instead it is registered as specific to the interval forecasting wrappers.

This also means that the evaluate specific tests in test_evaluate - which run if evaluate is changed - did not cover the failure, while they should.

FYI @hazrulakmal

[fkiraly]

oh, I remember what this is about - this is related to our earlier discussion that the pinball loss was capturing the alpha or coverage parameter for the predict method. We have not resolved that question, have we?

The problem is that previously, there was an internal hacky logic that passed on parameters attached to the metric to the evaluator, which now has been removed with the improved, streamlined interface.

That is, this used to work and no longer does, as the alpha are ignored:

evaluate(
        stuff,
        scoring=PinballLoss(alpha=[0.1, 0.5, 0.9]),
        more_stuff,
    )

There are two remaining problems:

• now we can no longer pass alpha to the metric to evaluate, we need to find a syntax before the next release
• the matter of deprecating the functionality, in case we decide to change the syntax

[fkiraly]

A salomonic option would be:

• add another argument to evaluate or deal with the matter another way sustainably
• should the metric have a fitting parameter or attribute, read it from there, until possibly deprecation

[fkiraly]

solved here: #5354