Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Performance Metric: multilevel="raw_values" with either multioutput="uniform_average" or custom weights for multioutput #6413

Closed
kdekker-private opened this issue May 13, 2024 · 2 comments · Fixed by #6418
Labels
bug Something isn't working module:metrics&benchmarking metrics and benchmarking modules
Projects

Comments

@kdekker-private
Copy link

Describe the bug
Performance metric classes do not work when using the combination multilevel="raw_values" with either multioutput="uniform_average" or custom weights for multioutput.

To Reproduce

Using MeanAbsoluteError, but the error does not remain to this class.

import numpy as np
from sktime.performance_metrics.forecasting import MeanAbsoluteError
y_true = np.array([[0.5, 1], [-1, 1], [7, -6]])
y_pred = np.array([[0, 2], [-1, 2], [8, -5]])
mae = MeanAbsoluteError(multilevel="raw_values", multioutput="uniform_average")
mae(y_true, y_pred)
mae_custom_multitoutput_weights = MeanAbsoluteError(multilevel="raw_values", multioutput=[0.4,0.6])
mae_custom_multitoutput_weights(y_true, y_pred)

Output:

ValueError: DataFrame constructor not properly called!

Expected behavior
Both the output of mae and mae_custom_multitoutput_weights with custom multioutput weights should not throw an error.

Additional context
It seems like the wrapper class around the inner metric function fails to coerse the evaluations in the expected output format.

Versions
0.28.0

@kdekker-private kdekker-private added the bug Something isn't working label May 13, 2024
@fkiraly fkiraly added the module:metrics&benchmarking metrics and benchmarking modules label May 13, 2024
@fkiraly fkiraly added this to Needs triage & validation in Bugfixing via automation May 13, 2024
@fkiraly
Copy link
Collaborator

fkiraly commented May 13, 2024

confirmed on current main, windows, python 3.11.

Note to devs: we need to check why this combination of inputs is not tested.

@fkiraly fkiraly moved this from Needs triage & validation to Reproduced/confirmed in Bugfixing May 13, 2024
@fkiraly
Copy link
Collaborator

fkiraly commented May 13, 2024

I am noting that the data in question has no hierarchy levels, as it is a "plain" series data container.

Still, the metric should not crash, so the coercion should be fixed.

@fkiraly fkiraly moved this from Reproduced/confirmed to Under review in Bugfixing May 14, 2024
fkiraly added a commit that referenced this issue May 19, 2024
…es is not hierarchical (#6418)

This PR allows metric classes to be called with `multilevel` arg in all
cases, if the series is not hierarchical. Previously, this would crash
the metric.

This PR changes behaviour in this degenerate case so a single-entry
`pd.DataFrame` is returned.

Fixes #6413
Bugfixing automation moved this from Under review to Fixed/resolved May 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working module:metrics&benchmarking metrics and benchmarking modules
Projects
Bugfixing
Fixed/resolved
2 participants