[ENH] rolling window (out of sample) detrender

[fkiraly]

It would be good to have a rolling window detrender that allows out-of-sample forecast detrending, i.e., computing rolling window out-of-sample residuals rather than in-sample residuals.

The current Detrender computes in-sample residuals, when naively applied (via fit_transform, which is called in a forecasting pipeline).

Not sure whether this should be an option in the class or a separate class - it should probably take a splitter (inheriting from BaseSplitter) as an argument. There is also some thought to be had around what should happen if the series seen in transform is not entirely identical nor entirely different from the one in fit.

If someone wants to take this up soon, it is perhaps best to base this on the simplified logic in #4053 rather than the current main.

[blazingbhavneek]

Hey! I would like to take it, seems a little intimidating to read, but can you suggest some files to read to get idea of what's happening? What i am trying to change?

[fkiraly]

Well, it's more of an extension than a change - there should be an option for "out of sample", and those are computed by the update_predict method (with a splitter inside).

I would recommend to read up on update_predict in the forecasting tutorial (part 1) and see how it compares to the current logic of Detrender which just uses predict.

[blazingbhavneek]

Hey! @fkiraly (sorry for the delay, I had exams in college)
I am new to these terminologies, so I had a hard time making sense, but as far as I could understand, I have to implement something like this, right?

Where the rolling window is adding new data with update_predict, correct? Or am I missing something?

[fkiraly]

Yes, exactly - and we use the union of the out-of-sample forecasts to detrend (subtract from the y), rather than in-sample forecasts.

[blazingbhavneek]

Hey! @fkiraly, I have a few questions:

1. What is the minimum size of the dataset required to train a forecaster initially? (To be on a safe side) Before we start to update_predict it.
2. What should be the fh? Should it depend on whether the forecaster requires fh or not?

[fkiraly]

1. What is the minimum size of the dataset required to train a forecaster initially? (To be on a safe side) Before we start to update_predict it.

There is no minimum that the interface enforces (i.e., 1 data point); most forecasters will break though if you start going below 10. The minimum may also be influenced by seasonality.

There is also an open design discussion here on how to handle the "minimum data requirement" in the interface: #3901
with @bethrice44 and @chillerobscuro, FYI

2. What should be the fh? Should it depend on whether the forecaster requires fh or not?

Hm, that's an interesting point. I would let the cv (splitter) determine fh, with the default case being a sliding window splitter that ingests data points one-by-one.

[blazingbhavneek]

Hey! @fkiraly Sorry for the delay, but I am facing some issues (forgive me if I missed something, I still have a lot to learn about the codebase)

1. Polynomial forecaster is giving me this warning

D:\Open Source Repos\sktime\sktime\forecasting\base_base.py:1882: UserWarning: NotImplementedWarning: PolynomialTrendForecaster does not have a custom update method implemented. PolynomialTrendForecaster will be refit each time update is called with update_params=True. To refit less often, use the wrappers in the forecasting.stream module, e.g., UpdateEvery.
warn(

2. If I set a rolling window size, its giving me a "cannot join overlapping dataframes", when I fixed this (and when I go with default window size, i.e 1 (default ExpandingWindowSplitter in update predict method)) it is returning a weird table with Nan values and almost double size.

Should I implement my own rolling window without using a splitter? It should be easier and hassle-free considering we only need a for loop? (although if I am not wrong about the issues above there might be extra discussion required for some already implemented classes)

[fkiraly]

@blazingbhavneek, no worries! We're a volunteer community and there is no expectation to be always engaged.

To answer your Q:

1. The warning is given by any forecaster that does not have its own update, it falls back to the default of refitting.

2. can you post some code that produces this? This does not seem right.

[blazingbhavneek]

@fkiraly I made a PR some time ago, and I am facing a issue, kindly check that out please.

[fkiraly]

oh, apologies, did not spot it! Somehow I didn't see the notification.