[BUG] StatsforecastMSTL - predict_quantile, predict_quantile, predict_interval not working

[EliasKng]

sktime/sktime/forecasting/base/_base.py

Line 598 in 3c40420

def predict_interval(self, fh=None, X=None, coverage=0.90):

Hi together,

every time I am trying to use one of the above functions, I get the same error. Adding quantiles.reducing fh to a single value instead of an array doesn't make a difference.

It also happens if I use the data provided in the examples.

Thanks a lot!
Cheers
Elias

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[57], line 1
----> 1 y_pred2 = fitted_model.predict_quantiles(fh=[i for i in range(1,90)]) 

File ~/Desktop/PTSFC/venv/lib/python3.9/site-packages/sktime/forecasting/base/_base.py:586, in BaseForecaster.predict_quantiles(self, fh, X, alpha)
    584 # we call the ordinary _predict_quantiles if no looping/vectorization needed
    585 if not self._is_vectorized:
--> 586     quantiles = self._predict_quantiles(fh=fh, X=X_inner, alpha=alpha)
    587 else:
    588     # otherwise we call the vectorized version of predict_quantiles
    589     quantiles = self._vectorize(
    590         "predict_quantiles",
    591         fh=fh,
    592         X=X_inner,
    593         alpha=alpha,
    594     )

File ~/Desktop/PTSFC/venv/lib/python3.9/site-packages/sktime/forecasting/base/_base.py:2081, in BaseForecaster._predict_quantiles(self, fh, X, alpha)
   2078     coverage = abs(1 - 2 * a)
   2080     # compute quantile forecasts corresponding to upper/lower
-> 2081     pred_a = self._predict_interval(fh=fh, X=X, coverage=[coverage])
   2082     pred_int = pd.concat([pred_int, pred_a], axis=1)
   2084 # now we need to subset to lower/upper depending
   2085 #   on whether alpha was < 0.5 or >= 0.5
   2086 #   this formula gives the integer column indices giving lower/upper

File ~/Desktop/PTSFC/venv/lib/python3.9/site-packages/sktime/forecasting/base/adapters/_generalised_statsforecast.py:348, in _GeneralisedStatsForecastAdapter._predict_interval(self, fh, X, coverage)
    345     interval_predictions.append(in_sample_interval_predictions)
    347 if out_of_sample_horizon:
--> 348     out_of_sample_interval_predictions = self._predict_in_or_out_of_sample(
    349         out_of_sample_horizon,
    350         "out-of-sample",
    351         X=X_predict_input,
    352         levels=coverage,
    353     )
    354     interval_predictions.append(out_of_sample_interval_predictions)
    356 final_interval_predictions = pandas.concat(interval_predictions, copy=False)

File ~/Desktop/PTSFC/venv/lib/python3.9/site-packages/sktime/forecasting/base/adapters/_generalised_statsforecast.py:189, in _GeneralisedStatsForecastAdapter._predict_in_or_out_of_sample(self, fh, fh_type, X, levels)
    186     # Before v1.5.0 (from statsforecast) not all foreasters
    187     # have a "level" keyword argument in `predict`
    188     level_kw = {"level": level_arguments} if self._support_pred_int else {}
--> 189     predictions = predict_method(maximum_forecast_horizon, X=X, **level_kw)
    190     point_predictions = predictions["mean"]
    192 if isinstance(point_predictions, pandas.Series):

File ~/Desktop/PTSFC/venv/lib/python3.9/site-packages/statsforecast/models.py:5011, in MSTL.predict(self, h, X, level)
   5009 level = sorted(level)
   5010 if self.trend_forecaster.prediction_intervals is not None:
-> 5011     res = self.trend_forecaster._add_predict_conformal_intervals(res, level)
   5012 else:
   5013     raise Exception(
   5014         "You have to instantiate either the trend forecaster class or MSTL class with `prediction_intervals` to calculate them"
   5015     )

AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'

[fkiraly]

Thanks for reporting!

Can you please provide us with a reproducible example that fails? I.e., python code, optimally using some dummy data?
I assume you are trying to work with StatsforecastMSTL?

[nestormontano]

Hi everybody,

I'm having the same error, here is a reproducible example:

import numpy as np # arrays
import pandas as pd # manejo de datos
from datetime import datetime #Para crear fechas
import sktime.split # Crea train-test/validacion
import sktime.forecasting.model_evaluation  
import sktime.utils.plotting # graficos
import sktime.forecasting.exp_smoothing # Modelos basados en Suavizacion Exponencial
import sktime.forecasting.ets # Modelos ETS statsmodels
import sktime.forecasting.statsforecast  # Modelos ETS statsforecast
import sktime.forecasting.trend # STL
import sktime.performance_metrics.forecasting as metricas # metricas

vuelos= pd.DataFrame({
    'MES': pd.date_range(start='2010-01-01', end='2015-12-01', freq='MS') ,
    'TOTAL': [8912, 8418, 9637, 9363, 9360, 9502, 9992, 10173, 9417, 9762, 9558, 9429, 9000, 8355, 9501, 9351, 9542, 9552, 9896, 9909, 8845, 9100, 8496, 8146, 8228, 8016, 8869, 8793, 8987, 8751, 8960, 9140, 8293, 8809, 8345, 8024, 8168, 7714, 9195, 9318, 9580, 9750, 10291, 10392, 9290, 9702, 9075, 8890, 8283, 7755, 9322, 9374, 9534, 9662, 10098, 9932, 9105, 9673, 9020, 8872, 8841, 8383, 9980, 10005, 10243, 10544, 10837, 10728, 9724, 10161, 9463, 9103]    
})

vuelos['MES'] = pd.to_datetime(vuelos['MES'], format= "%Y-%m-%d")
vuelos.set_index('MES', inplace=True)
vuelos.index = pd.PeriodIndex(vuelos.index, freq="M")

### MSTL con 12 y 18
m_mstl= sktime.forecasting.statsforecast.StatsForecastMSTL(season_length=[12,18]) 


m_mstl.fit( vuelos.TOTAL)


fh_rel = sktime.forecasting.base.ForecastingHorizon(np.arange(1, 13), is_relative= True)
fh_rel


m_mstl.predict(fh_rel)


m_mstl.predict_interval(coverage=0.9)

thanks in advance

[fkiraly]

Thanks for reporting!

Is this perhaps a version or inheritance issue, as it seems to be looking for a private method _add_predict_conformal_intervals?

[yarnabrina]

I am facing this now as part of #5920, where tests are failing with same error.

@sd2k it seems the test you added is causing the failures in our old CI, e.g. https://github.com/sktime/sktime/actions/runs/7848407801/job/21419578872#step:10:774. It's likely to be coming from the adapter itself, but if you have any idea what might by going wrong, that'd be very much appreciated.

[fkiraly]

@yarnabrina, which test are you referring to?

[yarnabrina]

Example from old CI (this is the one @sd2k added):

FAILED sktime/forecasting/tests/test_statsforecast.py::test_statsforecast_mstl - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'

Example from new CI:

FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-1-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-2-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-0-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-2-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-1-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-0-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-2-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-0-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-1-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-1-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_persistence_via_pickle[StatsForecastMSTL-0-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_save_estimators_to_file[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-2-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[StatsForecastMSTL-1-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-2-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-1-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_fit_idempotent[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-0-ForecasterFitPredictUnivariateWithX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-2-ForecasterFitPredictUnivariateWithX-predict_proba] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_non_state_changing_method_contract[StatsForecastMSTL-0-ForecasterFitPredictMultivariateNoX-predict_var] - AttributeError: 'StatsForecastBackAdapter' object has no attribute '_add_predict_conformal_intervals'

[fkiraly]

ah yes, these. I suppose this is something coming from a statsforecast base class that the back adapter has no access to?