[BUG] skip sporadic test errors in ExponentialSmoothing

[achieveordie]

Skips failures in #5026 by removing test case triggering it - does not solve the bug.

Due to the bug, ExponentialSmoothing's .predict() returns NaN sporadically, causing subsequent operations to fail since they expect finite values.

[fkiraly]

one question is also, does this happen locally as well as remote, or only remote?

Locally, you can run a loop of 100 check_estimator-s with the offending test and see how often it happens.

[achieveordie]

I tried what you're suggesting with only 10 iterations but found no errors. I'll try increasing it.

[fkiraly]

you can get more iterations in if you select the specific test

[achieveordie]

Here are my findings:

• The failing estimator is ExponentialSmoothing and the failing method is TestAllForecasters::test_score() (from the runners in the PR).
• Every single call yields 12 tests in total(different param combinations etc), here's an example of one of those:

{'test_score[ExponentialSmoothing-0-y:1cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-0-y:1cols-fh=[2 5]]': 'PASSED', 'test_score[ExponentialSmoothing-0-y:2cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-0-y:2cols-fh=[2 5]]': 'PASSED', 'test_score[ExponentialSmoothing-1-y:1cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-1-y:1cols-fh=[2 5]]': 'PASSED', 'test_score[ExponentialSmoothing-1-y:2cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-1-y:2cols-fh=[2 5]]': 'PASSED', 'test_score[ExponentialSmoothing-2-y:1cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-2-y:1cols-fh=[2 5]]': 'PASSED', 'test_score[ExponentialSmoothing-2-y:2cols-fh=1]': 'PASSED', 'test_score[ExponentialSmoothing-2-y:2cols-fh=[2 5]]': 'PASSED'}

• I ran the test for 1000 iterations getting 12000 total test runs, out of which 46 failed. Curiously, these failures were coming from only two combinations:

{'test_score[ExponentialSmoothing-2-y:1cols-fh=[2 5]]': 14, 'test_score[ExponentialSmoothing-2-y:2cols-fh=[2 5]]': 32}

(Note: ExponentialSmoothing-2 is the one that contains use_boxcox=True if the order of test params is considered the same order)

If my hypothesis is correct, this number should go down to zero after use_boxcox=False in test params. I'll soon update this comment after running the diagnosis script with new param set (will also add the script here for reference).

Edit: I can confirm that no tests fail after changing to use_boxcox=False. Here's my quick and dirty script that I've used, takes about 20 mins for 1000 iterations:

from sktime.utils.estimator_checks import check_estimator
from sktime.forecasting.exp_smoothing import ExponentialSmoothing
from tqdm import tqdm

ITERATIONS = 1000
result = []
for i in range(ITERATIONS):
    if i % 10 == 0:
        print("Iteration number: ", i)
    result.append(
        check_estimator(
            estimator=ExponentialSmoothing,
            tests_to_run="test_score",
            verbose=False
        )
    )

passed, failed = 0, 0
types_of_failures = {}

for i in result:
    for key, val in i.items():
        if val == "PASSED":
            passed += 1
        else:
            failed += 1
            types_of_failures[key] = types_of_failures.get(key, 0) + 1

print("=="*25)
print(passed, failed)
print("--"*25)
print(types_of_failures)

[yarnabrina]

I am not sure if it's a bugfix. It uses a test case that passes, but does it resolve the bug that occasionally fails the other test case(s)?

[achieveordie]

I am not sure if it's a bugfix. It uses a test case that passes, but does it resolve the bug that occasionally fails the other test case(s)?

This fixes the CI errors that were prominent across different PRs by "fixing" the test params. The actual source of error is considered to be numerical instability arising from statsmodel, and according to the comments, is by nature of this model (and perhaps their implementation).

So any other solution, if any, lies beyond the scope of sktime.

[fkiraly]

But that's not a "fix", in the sense that it just reduces test coverage away from the failure case? The failure case is still there, just no longer tested...

A "fix" would be sth, e.g., that does not reduce test coverage, but does sth to ensure we are interface compliant even in the sporadic failure case, such as filling zero, or restarting the fit, etc.

[achieveordie]

Hmm, I'm not sure if I agree with monkey-patching it. I agree with you and @yarnabrina (I think this is what he was implying too, I just misunderstood his statement) that this reduces the test coverage. Perhaps restarting the fit function should be a good alternative.

[fkiraly]

That wouldnt be a monkey-patch, would it? Simply a fallback patch in case it would have failed.