Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Gridsearch: Error metrics in split window mode - misleading order of method parameters. #979

Closed
ClaraGrthns opened this issue May 27, 2022 · 1 comment · Fixed by #989
Labels
bug Something isn't working triage Issue waiting for triaging

Comments

@ClaraGrthns
Copy link
Contributor

ClaraGrthns commented May 27, 2022

Describe the bug
In split window mode, the order of parameters is misleading. Normally, metrics like mape provided by darts get the actual series first and then the predicted series as input: darts.metrics.metrics.mape(actual_series, pred_series,...).

In the split-window mode of the gridsearch method, this is reversed, leading to incorrect results when comparing the different model configurations:

In darts.models.forecasting.forecasting_model.py (line 784):

@classmethod
    def gridsearch(
 ...

      else:  # split mode
                      model._fit_wrapper(series, past_covariates, future_covariates)
                      pred = model._predict_wrapper(
                          len(val_series),
                          series,
                          past_covariates,
                          future_covariates,
                          num_samples=1,
                      )
                      error = metric(pred, val_series)
      
                  return float(error)
...

Expected behavior
error = metric(val_series, pred).

System:

  • Python version: [e.g. 3.9.11]
  • darts version [e.g. 0.19.0]
@ClaraGrthns ClaraGrthns added bug Something isn't working triage Issue waiting for triaging labels May 27, 2022
@ClaraGrthns ClaraGrthns changed the title [BUG] Gridsearch - error metric, split window mode misleading method parameter order [BUG] Gridsearch: Error metrics in split window mode - misleading order of method parameters. May 27, 2022
@hrzn
Copy link
Contributor

hrzn commented May 27, 2022

Good catch! Do you want to open a PR with the fix you propose ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage Issue waiting for triaging
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants