Evaluation Metrics are missing and all models have failed, by an error in template or metrics

[tungsontran]

Describe the bug
I tried to use DNN models presented in the package such as GluonTS, pytorch-forecasting and tensorflowSTS. I have installed all required packages as in the installation guide (My environment can be viewed below). The error I got from the fit() function read as:

"Evaluation Metrics are missing and all models have failed, by an error in template or metrics. There are many possible causes for this, bad parameters, environment, or an unreported bug. Usually this means you are missing required packages for the models like fbprophet or gluonts, or that the models in model_list are inappropriate for your data. A new starting template may also help".

To Reproduce
When I run the following code:

from autots import AutoTS
model_list = ['TensorflowSTS']
model = AutoTS(
    forecast_length=7,
    frequency='D',
    prediction_interval=0.9,
    metric_weighting=metric_weighting,
    ensemble='horizontal',
    holiday_country='UK',
    model_list=model_list,  
    drop_most_recent=0,
    max_generations=1,
    num_validations=3,
    validation_method="backwards",
    no_negatives=True,
    verbose=-2,
    n_jobs = 72
)

model = model.fit(train_df)

Expected behavior
Stack trace:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/indexes/base.py:3621, in Index.get_loc(self, key, method, tolerance)
   3620 try:
-> 3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/_libs/index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/_libs/index.pyx:163, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5198, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5206, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'smape'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_model.py:2031, in generate_score(model_results, metric_weighting, prediction_interval)
   2030 # not sure why there are negative SMAPE values, but make sure they get dealt with
-> 2031 if model_results['smape'].min() < 0:
   2032     model_results['smape'] = model_results['smape'].where(
   2033         model_results['smape'] >= 0, model_results['smape'].max()
   2034     )

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/frame.py:3505, in DataFrame.__getitem__(self, key)
   3504     return self._getitem_multilevel(key)
-> 3505 indexer = self.columns.get_loc(key)
   3506 if is_integer(indexer):

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/indexes/base.py:3623, in Index.get_loc(self, key, method, tolerance)
   3622 except KeyError as err:
-> 3623     raise KeyError(key) from err
   3624 except TypeError:
   3625     # If we have a listlike key, _check_indexing_error will raise
   3626     #  InvalidIndexError. Otherwise we fall through and re-raise
   3627     #  the TypeError.

KeyError: 'smape'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File <timed exec>:44, in <module>

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_ts.py:791, in AutoTS.fit(self, df, date_col, value_col, id_col, future_regressor, weights, result_file, grouping_ids, validation_indexes)
    789 # capture the data from the lower level results
    790 self.initial_results = self.initial_results.concat(template_result)
--> 791 self.initial_results.model_results['Score'] = generate_score(
    792     self.initial_results.model_results,
    793     metric_weighting=metric_weighting,
    794     prediction_interval=prediction_interval,
    795 )
    796 if result_file is not None:
    797     self.initial_results.save(result_file)

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_model.py:2126, in generate_score(model_results, metric_weighting, prediction_interval)
   2123         overall_score = overall_score + (containment_score * containment_weighting)
   2125 except Exception as e:
-> 2126     raise KeyError(
   2127         f"""Evaluation Metrics are missing and all models have failed, by an error in template or metrics.
   2128         There are many possible causes for this, bad parameters, environment, or an unreported bug.
   2129         Usually this means you are missing required packages for the models like fbprophet or gluonts,
   2130         or that the models in model_list are inappropriate for your data.
   2131         A new starting template may also help. {repr(e)}"""
   2132     )
   2134 return overall_score.astype(float)

Desktop (please complete the following information):

• OS: Ubuntu 20.04
• Package Versions 0.4.2

``

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
absl-py 1.1.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.8.1 py310h5764c6d_1 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
alembic 1.8.0 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pypi_0 pypi
arch 5.3.1 pypi_0 pypi
argon2-cffi 21.3.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
asttokens 2.0.5 pypi_0 pypi
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 21.4.0 pyhd8ed1ab_0 conda-forge
autopage 0.5.1 pyhd8ed1ab_0 conda-forge
autots 0.4.2 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pypi_0 pypi
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.11.1 pypi_0 pypi
blas 1.0 mkl conda-forge
bleach 5.0.0 pypi_0 pypi
blinker 1.4 py_1 conda-forge
brotli 1.0.9 h166bdaf_7 conda-forge
brotli-bin 1.0.9 h166bdaf_7 conda-forge
brotlipy 0.7.0 py310h5764c6d_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2022.6.15 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 5.0.0 pyhd8ed1ab_0 conda-forge
certifi 2022.6.15 py310hff52083_0 conda-forge
cffi 1.15.0 py310h0fdd8cc_0 conda-forge
charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge
click 8.1.3 py310hff52083_0 conda-forge
cliff 3.10.1 pyhd8ed1ab_0 conda-forge
clikit 0.6.2 pypi_0 pypi
cloudpickle 2.1.0 pyhd8ed1ab_0 conda-forge
cmaes 0.8.2 pyh44b312d_0 conda-forge
cmd2 2.3.3 py310hff52083_1 conda-forge
cmdstanpy 0.9.5 pypi_0 pypi
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
colorlog 6.6.0 py310hff52083_1 conda-forge
convertdate 2.4.0 pypi_0 pypi
crashtest 0.3.1 pypi_0 pypi
cryptography 37.0.1 py310h9ce1e76_0
cudatoolkit 11.3.1 h9edb442_10 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cython 0.29.30 pypi_0 pypi
dask 2022.6.1 pypi_0 pypi
debugpy 1.6.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
dill 0.3.5.1 pypi_0 pypi
distributed 2022.6.1 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
ephem 4.1.3 pypi_0 pypi
executing 0.8.3 pypi_0 pypi
fastjsonschema 2.15.3 pypi_0 pypi
fonttools 4.33.3 pypi_0 pypi
freetype 2.10.4 h0708190_1 conda-forge
frozenlist 1.3.0 py310h5764c6d_1 conda-forge
fsspec 2022.5.0 pyhd8ed1ab_0 conda-forge
future 0.18.2 py310hff52083_5 conda-forge
gast 0.5.3 pyhd8ed1ab_0 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
gluonts 0.10.1 pypi_0 pypi
google-auth 2.9.0 pyh6c4a22f_0 conda-forge
google-auth-oauthlib 0.4.1 py_2 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
greenlet 1.1.2 py310hd8f1fbe_2 conda-forge
greykite 0.3.0 pypi_0 pypi
grpc-cpp 1.45.2 h3b8df00_4 conda-forge
grpcio 1.45.0 py310h44b9e0c_0 conda-forge
h5py 3.7.0 nompi_py310h06dffec_100 conda-forge
hdf5 1.12.1 nompi_h2386368_104 conda-forge
heapdict 1.0.1 pypi_0 pypi
hijri-converter 2.2.4 pypi_0 pypi
holidays 0.14.2 pypi_0 pypi
holidays-ext 0.0.7 pypi_0 pypi
httpstan 4.7.2 pypi_0 pypi
icu 70.1 h27087fc_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.4 py310hff52083_0 conda-forge
importlib_resources 5.8.0 pyhd8ed1ab_0 conda-forge
intel-openmp 2022.0.1 h06a4308_3633
ipykernel 6.15.0 pypi_0 pypi
ipython 8.4.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
jedi 0.18.1 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h166bdaf_2 conda-forge
jsonschema 4.6.0 pypi_0 pypi
jupyter-client 7.3.4 pypi_0 pypi
jupyter-console 6.4.4 pypi_0 pypi
jupyter-core 4.10.0 pypi_0 pypi
jupyter-http-over-ws 0.0.8 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
jupyterlab-widgets 1.1.1 pypi_0 pypi
keras 2.8.0 pyhd8ed1ab_0 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.3 py310hbf28c38_0 conda-forge
korean-lunar-calendar 0.2.1 pypi_0 pypi
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 14_linux64_mkl conda-forge
libbrotlicommon 1.0.9 h166bdaf_7 conda-forge
libbrotlidec 1.0.9 h166bdaf_7 conda-forge
libbrotlienc 1.0.9 h166bdaf_7 conda-forge
libcblas 3.9.0 14_linux64_mkl conda-forge
libcurl 7.83.1 h7bff187_0 conda-forge
libdeflate 1.12 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.1.0 h8d9b700_16 conda-forge
libgfortran-ng 12.1.0 h69a702a_16 conda-forge
libgfortran5 12.1.0 hdcd56e2_16 conda-forge
libgomp 12.1.0 h8d9b700_16 conda-forge
liblapack 3.9.0 14_linux64_mkl conda-forge
libnghttp2 1.47.0 h727a467_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge
libpng 1.6.37 h753d276_3 conda-forge
libprotobuf 3.20.1 h6239696_0 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge
libtiff 4.4.0 hc85c160_1 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp 1.2.2 h3452ae3_0 conda-forge
libwebp-base 1.2.2 h7f98852_1 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libzlib 1.2.12 h166bdaf_1 conda-forge
lightgbm 3.3.2 py310h122e73d_0 conda-forge
locket 1.0.0 pypi_0 pypi
lunarcalendar 0.0.9 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
mako 1.2.1 pyhd8ed1ab_0 conda-forge
markdown 3.3.7 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py310h5764c6d_1 conda-forge
marshmallow 3.16.0 pypi_0 pypi
matplotlib-base 3.5.2 py310h5701ce4_0 conda-forge
matplotlib-inline 0.1.3 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
mkl 2022.0.1 h06a4308_117
msgpack 1.0.4 pypi_0 pypi
multidict 6.0.2 py310h5764c6d_1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mxnet-cu112 1.9.1 pypi_0 pypi
nbclient 0.6.4 pypi_0 pypi
nbconvert 6.5.0 pypi_0 pypi
nbformat 5.4.0 pypi_0 pypi
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.5 pypi_0 pypi
notebook 6.4.12 pypi_0 pypi
numpy 1.23.0 py310h53a5b5f_0 conda-forge
oauthlib 3.2.0 pyhd8ed1ab_0 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1q h166bdaf_0 conda-forge
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
optuna 2.10.1 pyhd8ed1ab_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.3 py310h769672d_0 conda-forge
pandocfilters 1.5.0 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
partd 1.2.0 pypi_0 pypi
pastel 0.2.1 pypi_0 pypi
patsy 0.5.2 pyhd8ed1ab_0 conda-forge
pbr 5.9.0 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.1.1 pypi_0 pypi
pip 22.1.2 pyhd8ed1ab_0 conda-forge
pmdarima 1.8.5 pypi_0 pypi
prettytable 3.3.0 pypi_0 pypi
prometheus-client 0.14.1 pypi_0 pypi
prompt-toolkit 3.0.29 pypi_0 pypi
property-cached 1.6.4 pypi_0 pypi
prophet 1.1 pypi_0 pypi
protobuf 3.20.1 py310hd8f1fbe_0 conda-forge
psutil 5.9.1 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydantic 1.9.1 pypi_0 pypi
pydeprecate 0.3.2 pyhd8ed1ab_0 conda-forge
pygments 2.12.0 pypi_0 pypi
pyjwt 2.4.0 pyhd8ed1ab_0 conda-forge
pylev 1.4.0 pypi_0 pypi
pymeeus 0.5.11 pypi_0 pypi
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge
pyrsistent 0.18.1 pypi_0 pypi
pysimdjson 3.2.0 pypi_0 pypi
pysocks 1.7.1 py310hff52083_5 conda-forge
pystan 3.4.0 pypi_0 pypi
python 3.10.5 h582c2e5_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-flatbuffers 2.0 pyhd8ed1ab_0 conda-forge
python_abi 3.10 2_cp310 conda-forge
pytorch 1.12.0 py3.10_cuda11.3_cudnn8.3.2_0 pytorch
pytorch-forecasting 0.10.2 pyhd8ed1ab_0 conda-forge
pytorch-lightning 1.6.4 pyhd8ed1ab_0 conda-forge
pytorch-mutex 1.0 cuda pytorch
pytz 2022.1 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 5.4.1 pypi_0 pypi
pyzmq 23.2.0 pypi_0 pypi
qtconsole 5.3.1 pypi_0 pypi
qtpy 2.1.0 pypi_0 pypi
re2 2022.06.01 h27087fc_0 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rsa 4.8 pyhd8ed1ab_0 conda-forge
scikit-learn 1.1.1 py310hffb9edd_0 conda-forge
scipy 1.8.1 py310h7612f91_0 conda-forge
seaborn 0.11.2 pypi_0 pypi
send2trash 1.8.0 pypi_0 pypi
setuptools 59.5.0 py310hff52083_0 conda-forge
setuptools-git 1.2 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.9 hbd366e4_1 conda-forge
sortedcontainers 2.4.0 pypi_0 pypi
soupsieve 2.3.2.post1 pypi_0 pypi
sqlalchemy 1.4.39 py310h5764c6d_0 conda-forge
sqlite 3.38.5 h4ff8645_0 conda-forge
stack-data 0.3.0 pypi_0 pypi
statsmodels 0.13.2 py310hde88566_0 conda-forge
stevedore 3.5.0 py310hff52083_3 conda-forge
tblib 1.7.0 pypi_0 pypi
tensorboard 2.8.0 pyhd8ed1ab_1 conda-forge
tensorboard-data-server 0.6.0 py310h597c629_2 conda-forge
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
tensorflow 2.8.1 cpu_py310hd1aba9c_0 conda-forge
tensorflow-base 2.8.1 cpu_py310h17449b8_0 conda-forge
tensorflow-estimator 2.8.1 cpu_py310had6d012_0 conda-forge
termcolor 1.1.0 pyhd8ed1ab_3 conda-forge
terminado 0.15.0 pypi_0 pypi
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tinycss2 1.1.1 pypi_0 pypi
tk 8.6.12 h27826a3_0 conda-forge
toolz 0.11.2 pypi_0 pypi
torchmetrics 0.9.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 pypi_0 pypi
tqdm 4.64.0 pyhd8ed1ab_0 conda-forge
traitlets 5.3.0 pypi_0 pypi
typing-extensions 4.3.0 hd8ed1ab_0 conda-forge
typing_extensions 4.3.0 pyha770c72_0 conda-forge
tzdata 2022a h191b570_0 conda-forge
ujson 5.3.0 pypi_0 pypi
unicodedata2 14.0.0 py310h5764c6d_1 conda-forge
urllib3 1.26.9 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webargs 8.1.0 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
werkzeug 2.1.2 pyhd8ed1ab_1 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
widgetsnbextension 3.6.1 pypi_0 pypi
wrapt 1.14.1 py310h5764c6d_0 conda-forge
xgboost 1.6.1 pypi_0 pypi
xlrd 2.0.1 pypi_0 pypi
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.7.2 py310h5764c6d_2 conda-forge
zict 2.2.0 pypi_0 pypi
zipp 3.8.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.12 h166bdaf_1 conda-forge
zstd 1.5.2 h8a70e8d_2 conda-forge
``

[winedarksea]

Thanks for good bug report. Almost certainly TensorflowSTS is broken - it wasn't performing very well and I haven't maintained it in two years (note that it is in the 'Experimental' section of the model list). I will put it on my list to check into it.

I can guarantee that GluonTS and pytorch-forecasting are working on the latest release as I use them consistently.
model_list = ['GluonTS', 'PytorchForecasting', 'NeuralProphet', 'MultivariateRegression']
models_mode='neuralnets' if you want the `Regression models to only use Tensorflow, although it actually does better with non-neural net models, as is generally the case.

In my experience GluonTS is the best of the lot, so focus on that.

python -m pip install mxnet --no-deps     # check the mxnet documentation for more install options, also try pip install mxnet --no-deps
python -m pip install gluonts