[ENH] HierarchyEnsembleForecaster for level- or node-wise application of forecasters on panel/hierarchical data

[VyomkeshVyas]

Reference Issues/PRs

Fixes #2764

What does this implement/fix? Explain your changes.

HierarchyEnsembleForecaster() aggregates panel-type data and applies different univariate forecasters on the aggregated data by each hierarchical level/node. For aggregation, it employs sktime's bulit-in 'Aggregator' class.

Does your contribution introduce a new dependency? If yes, which one?

No

What should a reviewer concentrate their feedback on?

A reviewer should concentrate their feedback on the forecaster's ability to :-

• Fit a separate forecaster on each hierarchical level of the panel/hierarchical data, with and without exogenous data.
• Fit a separate forecaster on each hierarchical node of the panel/hierarchical data, with and without exogenous data.
• Fit a 'default' forecaster (if passed as argument) on nodes/levels not mentioned in the 'forecasters' argument.
• Make predictions by each hierarchical level of the fitted aggregated panel data.
• Make predictions by each hierarchical node of the fitted aggregated panel data.

Any other comments?

PR checklist

For all contributions

• I've added myself to the list of contributors.
• Optionally, I've updated sktime's CODEOWNERS to receive notifications about future changes to these files.
• I've added unit tests and made sure they pass locally.
• The PR title starts with either [ENH], [MNT], [DOC], or [BUG] indicating whether the PR topic is related to enhancement, maintenance, documentation, or bug.

For new estimators

• I've added the estimator to the online documentation.
• I've updated the existing example notebooks or provided a new one to showcase how my estimator works.

[fkiraly]

neat! let us know when you would like a review

[VyomkeshVyas]

neat! let us know when you would like a review

Hi @fkiraly, this PR is ready for review now. Thanks.

[fkiraly]

excellent! Will start the CI and we'll see if anything fails.

[fkiraly]

the doctest is failing, since it checks actual printout against expected.

When you run the line
>>> forecaster.fit(y, fh=[1, 2, 3]),

the printout is HierarchyEnsembleForecaster(

You can catch that by adding the line

HierarchyEnsembleForecaster(...) (like this, with three dots, and without three >), directly after

[fkiraly]

Great, looks good!

Some small things:

• can you kindly fix the doctest? Explanation is above.
• there is a merge conflict with the contributors file, kindly fix

[fkiraly]

hm, these look like genuine failures when you are trying to hash your node dict

[fkiraly]

recommendation: run your tests locally!

[VyomkeshVyas]

recommendation: run your tests locally!

Its strange but the tests didn't fail locally. Fixed the doctsring error, but still not able to recreate the unhashable type failure.

[fkiraly]

how odd. Should we try to run them again? I'll restart.

There is also a merge conflict in the contributors file, kindly update from main.

[fkiraly]

These are genuine failures now.

Have you run check_estimator locally and tried to debug?

I would recommend that, see here:
https://www.sktime.org/en/stable/developer_guide/add_estimators.html

[VyomkeshVyas]

These are genuine failures now.

Have you run check_estimator locally and tried to debug?

I would recommend that, see here: https://www.sktime.org/en/stable/developer_guide/add_estimators.html

Hi @fkiraly, I have the updated the code and majority of the issues are fixed. But still, some tests (around 12) are failing due to a common error :- ValueError('Length of names must match number of levels in MultiIndex.').

This error is inevitable for a particular test instance, the way the check_estimator tests are designed currently. Why I think so?
I'll give a very brief overview of HierarchyEnsemblerForecaster(). The hier-ensm forecaster first aggregates the data and then fits a separate forecaster either by level or by nodes. For that, it takes three arguments : 'by', 'forecasters' and 'default'.
'by' can be 'level' or 'node', 'forecasters' can be list of tuples (name, BaseForecaster, level/node) or BaseForecaster and 'default' is BaseForecaster (which is None if not specified).

The above error is linked with all the test instances with by = 'node'. Ideally, the length of a particular node being passed in 'forecasters' attributes should be N-1, where N is the levels of multiindex data. (The last level of the data is assumed to be a timepoint index and hence N-1). The way the tests are designed, the hier-ensm forecaster will require different length of nodes being passed in 'forecasters' argument for different category of data (for eg univariate and panel data). Since, nodes can only be specified once for a particular category of data before running the tests, it fails for the other categories.

Could you please give some suggestions on how to handle this issue?

[fkiraly]

@VyomkeshVyas, sorry for the delay in the reply. I was fixing some merge conflicts and doctest errors in this PR so the tests would run through to the failures that you are referring to (hope that's ok). Once I can see the failures, I'll have a look.

[fkiraly]

Could you please give some suggestions on how to handle this issue?

Thanks for the explanation!
I think this is precisely the issue. It's a specific instance where the forecaster requires assumptions on the data format that are stronger than the input contract.

Indeed, as the test framework is designed, all inputs are passed to all forecasters, which in this case upsets the new forecaster, as its parameters need to match the levels.

I see multiple solutions:

• test the node case separately, not using the default framework. I.e., skip the appropriate framework tests by not including the parameters in get_test_params or skipping tests via tests/_config, and add manual tests instead
• modify the forecaster so it does something sensible for data that doesn't match the node specification (e.g., nodes not present are simply ignored or similar)
• an extension to the testing framework or estimators that allows compatibility checks between parameters and data - I have been thinking about this but this would require a bigger design (e.g., STEP), written by me, you or someone else, so it may be overkill for the problem at hand

[VyomkeshVyas]

@fkiraly Thanks for the suggestions! That's very helpful.
I have added a new functionality for a test instance when length of individual node being passed mismatches the level of multi-index data. The fix seems to work well as all the tests are passed now but, I would like to have your opinion whether that's a right solution. For example, if I have a data with multi-index (A,B,C,D) with D being the timepoint index and 'forecasters' being passed is ('f', F, [(x, y)] ). Then, the forecaster F will be fitted to the data with multi-index (A, B) == (x, y), which previously would have required (A, B, C) == (x, y, z).

[VyomkeshVyas]

I forgot to add aggregation levels in X, y in update and predict functions. I am working on it.

[fkiraly]

i.e., you went with option 2, right?
Makes sense, with the "ignore nodes" option.

I forgot to add aggregation levels in X, y in update and predict functions. I am working on it.

Let us know when you think this is ready.

[VyomkeshVyas]

i.e., you went with option 2, right? Makes sense, with the "ignore nodes" option.

yea, with option 2. But, instead I am not ignoring the mismatched node, rather grouping all the nodes which are super set of mismatch node.

I forgot to add aggregation levels in X, y in update and predict functions. I am working on it.

Let us know when you think this is ready.

Its ready now. Thanks again.

[fkiraly]

Excellent! Impressive for a first contribution!

To be frank, when I saw the first version and the specs, I thought, "well this might end in sweat and tears", because it was very ambitious - hierarchical data, _HeterogenousMetaEstimator which is not easy to inherit from, the issue with "data must fit the parameters" which I also didn't fully see how to solve, a docstring which is difficult to write with formal accuracy, etcetera.

But none of that blocked you!
Absolutely impressive!!

Welcome to sktime, @VyomkeshVyas!
Way to make an entrance.

[VyomkeshVyas]

@fkiraly Thank you very much !!
It's been a great learning for me and I totally enjoyed it. A big shout out to @ciaran-g for continuous guidance, without which it might actually have ended in "sweat and tears".

[fkiraly]

A big shout out to @ciaran-g for continuous guidance, without which it might actually have ended in "sweat and tears".

Well, that's why sktime is a community of contributors - to help reach the best of one's potential :-)