Handle prediction intervals in `TSDataset` similar to target components #97

brsnw250 · 2023-10-04T11:32:40Z

Before submitting (must do checklist)

Did you read the contribution guide?
Did you update the docs? We use Numpy format for all the methods and classes.
Did you write any new necessary tests?
Did you update the CHANGELOG?

Proposed Changes

Closing issues

closes #88

github-actions · 2023-10-04T11:39:50Z

🚀 Deployed on https://deploy-preview-97--etna-docs.netlify.app

codecov · 2023-10-04T12:04:56Z

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (be58b43) 89.14% compared to head (d9565c1) 89.18%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #97      +/-   ##
==========================================
+ Coverage   89.14%   89.18%   +0.04%     
==========================================
  Files         195      195              
  Lines       12554    12551       -3     
==========================================
+ Hits        11191    11194       +3     
+ Misses       1363     1357       -6

Files	Coverage Δ
etna/models/mixins.py	`96.39% <100.00%> (+0.13%)`	⬆️
etna/models/nn/deepar.py	`97.40% <100.00%> (-0.10%)`	⬇️
etna/models/nn/tft.py	`97.87% <100.00%> (-0.07%)`	⬇️
etna/pipeline/base.py	`95.72% <100.00%> (+0.01%)`	⬆️
etna/pipeline/hierarchical_pipeline.py	`95.18% <100.00%> (+0.11%)`	⬆️
etna/reconciliation/base.py	`95.23% <100.00%> (+0.64%)`	⬆️
etna/transforms/base.py	`92.85% <100.00%> (-0.05%)`	⬇️
...forms/decomposition/change_points_based/detrend.py	`100.00% <100.00%> (ø)`
etna/transforms/decomposition/detrend.py	`96.00% <100.00%> (-0.30%)`	⬇️
etna/transforms/decomposition/stl.py	`93.33% <100.00%> (-0.34%)`	⬇️
... and 7 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

etna/datasets/tsdataset.py

d-a-bunin · 2023-10-05T08:53:11Z

etna/transforms/base.py

@@ -208,7 +208,7 @@ def __init__(self, required_features: Union[Literal["all"], List[str]]):
        super().__init__(required_features=required_features)

    @abstractmethod
-    def _inverse_transform(self, df: pd.DataFrame) -> pd.DataFrame:
+    def _inverse_transform(self, df: pd.DataFrame, prediction_intervals: Tuple[str, ...]) -> pd.DataFrame:


Can't we somehow avoid changing the signature in backward incompatible way?

We can make this field optional, but it requires doing the same check in many different places.
But how does this make changes backward incompatible? We make this change only in internal private methods and classes that are not intended for external use cases.

A small follow up, we can do it without the check at all.
The question is still relevant.

Imagine that someone created their own transforms. With these changes their transforms stop working and need rework to support new way of handling intervals.

With models we still use match_target_quantiles for extracting quantiles from adapters. Can't we make smth like this for transforms too?

I see.

Unfortunately, we can't use match_target_quantiles, because transforms should work with any interval borders, not only quantilies, so they might not follow name convention. In this case, we should propagate interval names from the top level (where we work with datasets). Built-in models intervals produce quantiles, that's why we can use this function there.

d-a-bunin · 2023-10-05T08:56:24Z

etna/transforms/math/limit.py

@@ -114,13 +115,15 @@ def _transform(self, df: pd.DataFrame) -> pd.DataFrame:

        return transformed_features

-    def _inverse_transform(self, df: pd.DataFrame) -> pd.DataFrame:
+    def _inverse_transform(self, df: pd.DataFrame, prediction_intervals: Tuple[str, ...]) -> pd.DataFrame:


Doesn't we have to handle prediction intervals in this transform too?

This transform handles all columns of df similarly. Intervals are stored in df directly, so they are transformed as well.

d-a-bunin · 2023-10-05T09:04:45Z

tests/test_datasets/test_dataset.py

+
+@pytest.fixture()
+def ts_with_prediction_intervals(ts_without_target_components, prediction_intervals_df):
+    ts = deepcopy(ts_without_target_components)


I'm not sure that it is necessary to make deepcopy in fixtures.

If we don't do so, this fixture will modify ts_without_target_components inplace. But we want it untouched.

I thought that fixtures doesn't interact with each other during creation.

tests/test_datasets/test_dataset.py

tests/test_models/nn/test_deepar.py

tests/test_pipeline/test_hierarchical_pipeline.py

d-a-bunin · 2023-10-09T11:52:10Z

etna/datasets/tsdataset.py

@@ -515,9 +525,17 @@ def target_components_names(self) -> Tuple[str, ...]:
        return self._target_components_names

    @property
+    @deprecated(
+        reason="Usage of this property may mislead while accessing prediction intervals. Use `prediction_intervals_names` property to access intervals names!"


Don't we want to set a version in which it is going to be removed?

Idealy, it should be removed when new prediction intervals are completely stable and moved from experimental. Right now, it seems like an open question.

We can't remove this property without major update. So, let's set version 3.0 for example for this property to be removed.

d-a-bunin · 2023-10-09T12:02:04Z

etna/transforms/math/sklearn.py

-            for quantile_column_nm in quantiles:
+            # inverse transformation of columns that are left (e.g. prediction intervals)
+            other_columns = set(df.columns.get_level_values("feature")) - set(self.in_column)
+            for column_nm in other_columns:


Could we use column_name?

brsnw250 added 19 commits September 29, 2023 10:42

handle prediction intervals in TSDataset

96e35cf

updated fixtures

4865ac6

reworked fixture

6cfafb5

added tests

da8b9c8

new prediction intervals logic in reconcile

3201993

updated tests

7497109

formatting

45d8855

formatted test

e2b1bab

updated DeepAR

a520027

updated tests for DeepAR

3a55015

updated prediction intervals for TFT

6d5dc9f

updated tests

094d97c

updated PerSegmentModelMixin

651f896

updated model specific intervals tests

fd75a42

added prediction intervals names to inverse_transform signature

d03f503

use new functionality in reversible transforms

be8a19e

updated tests

4de3997

updated interval store in pipeline

4407853

updated HierarchicalPipeline

33a8eb8

brsnw250 self-assigned this Oct 4, 2023

github-actions bot temporarily deployed to pull request October 4, 2023 11:39 Inactive

brsnw250 added 2 commits October 4, 2023 14:56

formatting

b915fa0

updated changelog

484a7aa

brsnw250 changed the title ~~Issue 88~~ Handle prediction intervals in TSDataset similar to target components Oct 4, 2023

github-actions bot temporarily deployed to pull request October 4, 2023 12:05 Inactive

brsnw250 marked this pull request as ready for review October 5, 2023 06:43

d-a-bunin self-requested a review October 5, 2023 07:43

d-a-bunin requested changes Oct 5, 2023

View reviewed changes

brsnw250 added 2 commits October 5, 2023 13:34

review fixes

3dbe592

reworked test

325e6f8

github-actions bot temporarily deployed to pull request October 5, 2023 10:41 Inactive

brsnw250 requested a review from d-a-bunin October 5, 2023 10:58

brsnw250 added 2 commits October 9, 2023 13:47

reworked inverse transforms

67f3f22

updated tests

1e88bbc

github-actions bot temporarily deployed to pull request October 9, 2023 10:53 Inactive

d-a-bunin reviewed Oct 9, 2023

View reviewed changes

d-a-bunin requested changes Oct 9, 2023

View reviewed changes

review fixes

3f7d230

brsnw250 requested a review from d-a-bunin October 9, 2023 12:24

github-actions bot temporarily deployed to pull request October 9, 2023 12:30 Inactive

d-a-bunin previously approved these changes Oct 9, 2023

View reviewed changes

Merge branch 'master' into issue-88

1ad0a13

github-actions bot temporarily deployed to pull request October 9, 2023 12:40 Inactive

Merge branch 'master' into issue-88

d9565c1

brsnw250 dismissed d-a-bunin’s stale review via d9565c1 October 9, 2023 14:07

github-actions bot temporarily deployed to pull request October 9, 2023 14:24 Inactive

d-a-bunin approved these changes Oct 9, 2023

View reviewed changes

brsnw250 merged commit 799ccb1 into master Oct 9, 2023
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle prediction intervals in `TSDataset` similar to target components #97

Handle prediction intervals in `TSDataset` similar to target components #97

brsnw250 commented Oct 4, 2023 •

edited

github-actions bot commented Oct 4, 2023 •

edited

codecov bot commented Oct 4, 2023 •

edited

d-a-bunin Oct 5, 2023

brsnw250 Oct 5, 2023

brsnw250 Oct 5, 2023

d-a-bunin Oct 5, 2023

brsnw250 Oct 5, 2023

d-a-bunin Oct 5, 2023

brsnw250 Oct 5, 2023

d-a-bunin Oct 5, 2023

brsnw250 Oct 5, 2023

d-a-bunin Oct 5, 2023

d-a-bunin Oct 9, 2023

brsnw250 Oct 9, 2023

d-a-bunin Oct 9, 2023

d-a-bunin Oct 9, 2023

Handle prediction intervals in TSDataset similar to target components #97

Handle prediction intervals in TSDataset similar to target components #97

Conversation

brsnw250 commented Oct 4, 2023 • edited

Before submitting (must do checklist)

Proposed Changes

Closing issues

github-actions bot commented Oct 4, 2023 • edited

codecov bot commented Oct 4, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Handle prediction intervals in `TSDataset` similar to target components #97

Handle prediction intervals in `TSDataset` similar to target components #97

brsnw250 commented Oct 4, 2023 •

edited

github-actions bot commented Oct 4, 2023 •

edited

codecov bot commented Oct 4, 2023 •

edited