[BUG] LSTM deep learning estimators failing CI on windows

[fkiraly]

Since recently, a failure of two deep learning estimators has been appearing on windows CI:

FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[MLPClassifier-1-ClassifierFitPredictMultivariate-predict]
FAILED sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[LSTMFCNClassifier-0-ClassifierFitPredictMultivariate-predict]

Note that the failure appears to be only on python 3.9, but this is due to the matrix design which spreads estimators across version/OS combinations.

In theory, it could be a 3.9 specific failure, but I think that is less likely than windows specifity (although it may be worth to test that by turning the matrixdesign flag off in the CI).

[fkiraly]

This does not seem clearly related to any recent PR merge to main.

@achieveordie, @AurumnPegasus, @jnrusson1, @solen0id, do you have a clue what is happening here?

[fkiraly]

traceback is not very informative:

 ================================== FAILURES ===================================
_____________________ sktime/tests/test_all_estimators.py _____________________
[gw0] win32 -- Python 3.9.12 C:\Miniconda\envs\test\python.exe
worker 'gw0' crashed while running 'sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[MLPClassifier-1-ClassifierFitPredictMultivariate-predict]'
_____________________ sktime/tests/test_all_estimators.py _____________________
[gw1] win32 -- Python 3.9.12 C:\Miniconda\envs\test\python.exe
worker 'gw1' crashed while running 'sktime/tests/test_all_estimators.py::TestAllEstimators::test_methods_have_no_side_effects[LSTMFCNClassifier-0-ClassifierFitPredictMultivariate-predict]'

[achieveordie]

LSTMFCN was added last week, right? I suspect the problem is arising from that estimator.

We have seen similar problems back when TapNet was ported, OutOfMemory/MemoryError error arising from one worker can crash the entire test suite. Hence we see MLPClassifier failing, just because it happened to be tested on the same system.

The reason I think LSTMFCN is causing the failures is that its get_test_params() currently only reduces the number of epochs, the size of the model remains the same.

Let me take this up and diagnose if there are any more problems in it.

[fkiraly]

Yes, it seems to be LSTMFCN - it also pops up in other failures recently.