Feature/historical retrain on condition #1139

FBruzzesi · 2022-08-10T12:22:53Z

Addresses #135 and #623

Summary

Implemented support for new types in retrain argument of historical_forecasts method. Now it accepts bool, (positive) int and Callable (returning a bool) data types.

The new behaviour is as follows (copying from the updated docstring):

In the case of bool: retrain the model at each step (True), or never retrains the model (False).
Not all models support setting retrain to False.
Notably, this is supported by neural networks based models.
In the case of int: the model is retrained every retrain iterations.
In the case of Callable: the model is retrained whenever callable returns True.
Notice that the arguments passed to the callable are as follows:
- pred_time (pd.Timestamp): next timestamp to predict (retraining happens before)
- series (TimeSeries): train series up to pred_time
- past_covariates (TimeSeries): past_covariates series up to pred_time
- future_covariates (TimeSeries): future_covariates series up to pred_time + series.freq * forecast_horizon

Other Information

In order to achive this behaviour, I also added a decorator function, called _retrain_wrapper, in utils/utils.py, which passes to the wrapped function only the original signature arguments, and raises a ValueError in the case that the provided function doesn't return a boolean value.
To test the behaviour, I wrote a test, called test_backtest_retrain, in test_local_forecasting_models.py as suggested in #623

…import

hrzn

Looks quite good, thanks! I have a couple of comments.

hrzn · 2022-08-10T14:17:04Z

darts/models/forecasting/forecasting_model.py

+            In the case of ``int``: the model is retrained every `retrain` iterations.
+            In the case of ``Callable``: the model is retrained whenever callable returns `True`.
+            Notice that the arguments passed to the callable are as follows:
+                - `pred_time (pd.Timestamp)`: next timestamp to predict (retraining happens before)


Suggested change

- `pred_time (pd.Timestamp)`: next timestamp to predict (retraining happens before)

- `pred_time (pd.Timestamp or int)`: timestamp of forecast time (end of the training series)

hrzn · 2022-08-10T14:18:56Z

darts/models/forecasting/forecasting_model.py

+            In the case of ``Callable``: the model is retrained whenever callable returns `True`.
+            Notice that the arguments passed to the callable are as follows:
+                - `pred_time (pd.Timestamp)`: next timestamp to predict (retraining happens before)
+                - `series (TimeSeries)`: train series up to `pred_time`


How about calling it train_series ?

darts/models/forecasting/forecasting_model.py

hrzn · 2022-08-10T14:45:26Z

darts/models/forecasting/forecasting_model.py

+            This parameter supports 3 different datatypes: ``bool``, (positive) ``int``, and
+            ``Callable`` (returning a ``bool``).
+            In the case of ``bool``: retrain the model at each step (`True`), or never retrains the model (`False`).
+            Not all models support setting `retrain` to `False`.


Could you move these two sentences down in the retrain doc? And slightly modify it, e.g.,

Note: some models do require being retrained every time and do not support anything else than `retrain=True`.

hrzn · 2022-08-10T14:49:38Z

darts/tests/models/forecasting/test_local_forecasting_models.py

@@ -336,3 +338,43 @@ def test_statsmodels_dual_models(self):
            # check backtesting with retrain=False
            model: TransferableDualCovariatesForecastingModel = model_cls(**kwargs)
            model.backtest(series1, future_covariates=exog1, retrain=False)
+
+    def test_backtest_retrain(self):


You could also add a test to make sure retrain is being called as expected (with expected arguments, e.g., past/future covariates etc) when it is a Callable. I think this should be pretty easy to do with unittest.mock, you can check out examples here: https://docs.python.org/3/library/unittest.mock.html

I wasn't able to use Mock, yet implemented a test with "fake" train_series, past_covariates and future_covariates all equal to the series itself, but testing on some "proper" condition.

This can also work, however your current test does not test correctness, or even that the function has been called. If you want to do it this way (using conditions on the series within the function), I would maybe cook a small toy example where the result of the backtest is not the same depending (for instance) on the quantile quantities computed on the covariates series. This way you could then test that

The output of the backtesting procedure is not the same if different future_covariates are used

The output is as you expect in some pre-computed case (e.g., the historical forecasts is a series containing values [x, y, z]).

That being said, I still think it might be easier to simply test whether the function is called by using mocking. You can find an example of how we are doing it in other tests, for instance here, where we then check here whether a given function has been called or not.

Just to explain myself: it's easy to mock a retrain function (i.e. a callable returning a boolean), but the function itself is called at each iteration (and I am able to check for that).
I would be more interested to verify that _fit_wrapper method is called multiple times.
However when trying to patch/mock _fit_wrapper and _predict_wrapper, I get all sort of errors.

@hrzn do you have any new suggestion on this?

Then if mocking does not work my suggestion would be to at least test that the output of model_cls.backtest() is what you expect it to be for a couple of simple cases (e.g., when you retrain each 1st of the month).

Finally I was able to make the test run using mock.patch: few tricks were needed. Looking forward to the review.

…drop_after' calls, added test with past and future covariates

codecov-commenter · 2022-08-11T20:09:50Z

Codecov Report

Base: 93.61% // Head: 93.61% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (bd6a92e) compared to base (4a27edd).
Patch coverage: 88.23% of modified lines in pull request are covered.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1139   +/-   ##
=======================================
  Coverage   93.61%   93.61%           
=======================================
  Files          81       81           
  Lines        8328     8330    +2     
=======================================
+ Hits         7796     7798    +2     
  Misses        532      532

Impacted Files	Coverage Δ
darts/utils/__init__.py	`100.00% <ø> (ø)`
darts/models/forecasting/forecasting_model.py	`96.49% <87.50%> (+0.39%)`	⬆️
darts/utils/utils.py	`93.51% <88.88%> (-0.43%)`	⬇️
darts/timeseries.py	`92.23% <0.00%> (-0.07%)`	⬇️
...arts/models/forecasting/torch_forecasting_model.py	`87.45% <0.00%> (-0.05%)`	⬇️
darts/models/forecasting/block_rnn_model.py	`98.24% <0.00%> (-0.04%)`	⬇️
darts/models/forecasting/nhits.py	`98.55% <0.00%> (-0.02%)`	⬇️
darts/datasets/__init__.py	`100.00% <0.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

hrzn

Thanks, the tests look better with the mocking!
There were a couple more comments to address and then I think we will be able to merge. Also please solve the merge conflicts first.

hrzn · 2022-08-23T09:32:14Z

darts/tests/models/forecasting/test_local_forecasting_models.py

+                    # resets patch_retrain_func call_count to 0
+                    retrain.call_count = 0
+                    retrain.side_effect = [True, False] * (len(series) // 2)
+                    # retrain.return_value = True


this line can be removed

Added os, shutil and tempfile imports

hrzn

Looks good! I just have 4 tiny commit suggestions and then I think we'll be good.

darts/models/forecasting/forecasting_model.py

Co-authored-by: Julien Herzen <j.herzen@gmail.com>

…del.py Co-authored-by: Julien Herzen <j.herzen@gmail.com>

hrzn

Thanks @FBruzzesi ! I'll merge it :)

FBruzzesi added 4 commits August 9, 2022 11:59

ihistorical retrain on condition

121cdd5

ihistorical retrain on condition

f7e6c2b

fixed raise_if logic at ln 396

2d8fb9e

added test in local_forecasting_models, changed decorator name hence …

6da635c

…import

FBruzzesi requested review from hrzn, tomasvanpottelbergh, dennisbader and brunnedu as code owners August 10, 2022 12:22

hrzn reviewed Aug 10, 2022

View reviewed changes

fixed retrain docstring, added check for retrain signature to avoid '…

2f2dd69

…drop_after' calls, added test with past and future covariates

add docstring to wrapper, make test run using mock patch

5fc63f2

hrzn reviewed Aug 23, 2022

View reviewed changes

FBruzzesi and others added 5 commits August 23, 2022 12:31

removed commented command in test and redundant assert statements

bba8921

Merge branch 'master' into feature/historical-retrain-on-condition

f10d6a2

deleted unused imports

8b9438a

added imports in test_local_forecasting_models

51a6873

Added os, shutil and tempfile imports

Merge branch 'master' into feature/historical-retrain-on-condition

410b3ca

hrzn reviewed Aug 26, 2022

View reviewed changes

FBruzzesi and others added 5 commits August 26, 2022 15:23

Update darts/models/forecasting/forecasting_model.py

df278ea

Co-authored-by: Julien Herzen <j.herzen@gmail.com>

Update docstring as suggested darts/models/forecasting/forecasting_mo…

274a07d

…del.py Co-authored-by: Julien Herzen <j.herzen@gmail.com>

Update docstring as suggested darts/models/forecasting/forecasting_mo…

fb8c143

…del.py Co-authored-by: Julien Herzen <j.herzen@gmail.com>

Update docstring as suggested darts/models/forecasting/forecasting_mo…

f995f41

…del.py Co-authored-by: Julien Herzen <j.herzen@gmail.com>

reformatted using black

bd6a92e

hrzn approved these changes Aug 30, 2022

View reviewed changes

Merge branch 'master' into feature/historical-retrain-on-condition

ebdd949

hrzn merged commit 4a522a0 into unit8co:master Aug 30, 2022

deltamacht mentioned this pull request Oct 31, 2022

backtest(): Retrain every n steps #135

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/historical retrain on condition #1139

Feature/historical retrain on condition #1139

FBruzzesi commented Aug 10, 2022

hrzn left a comment

hrzn Aug 10, 2022

hrzn Aug 10, 2022

hrzn Aug 10, 2022

hrzn Aug 10, 2022

FBruzzesi Aug 11, 2022

hrzn Aug 11, 2022

FBruzzesi Aug 12, 2022

FBruzzesi Aug 22, 2022

hrzn Aug 22, 2022

FBruzzesi Aug 23, 2022 •

edited

codecov-commenter commented Aug 11, 2022 •

edited

hrzn left a comment •

edited

hrzn Aug 23, 2022

hrzn left a comment

hrzn left a comment

	- `pred_time (pd.Timestamp)`: next timestamp to predict (retraining happens before)
	- `pred_time (pd.Timestamp or int)`: timestamp of forecast time (end of the training series)

Feature/historical retrain on condition #1139

Feature/historical retrain on condition #1139

Conversation

FBruzzesi commented Aug 10, 2022

Summary

Other Information

hrzn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FBruzzesi Aug 23, 2022 • edited

Choose a reason for hiding this comment

codecov-commenter commented Aug 11, 2022 • edited

Codecov Report

hrzn left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hrzn left a comment

Choose a reason for hiding this comment

hrzn left a comment

Choose a reason for hiding this comment

FBruzzesi Aug 23, 2022 •

edited

codecov-commenter commented Aug 11, 2022 •

edited

hrzn left a comment •

edited