Adding Deep Learning Support (#134)

* initial deep learning support * remove things * remove run vae * add deep forecast config * add autoformer and informer * add autoformer and informer * add transformer implementation * add evaluations * add normalization * add and modifying etsformer * remove uncessary arguments * modifying etsformer tests * finalizing deep learning parts, left document staff * Changing to dataset downloading using gdown * Delete auto-generated file. * move gdown library * change code design based on suggestions * Delete weather.csv * remove logging entire config * remove uncessary logging * remove uncessary files * remove device from Config class to TorchModel class * change setup.py to match requirement of main repo * fix device allocation typo * change timefeature file name * modify and test model loading and saving with map_location * update documentation and change various places for merging request * fix function calling bug * remove unecessary comments * update doc * fix typo * fix doc build typo * update doc * fix doc * change model call API to model(past, past_timestamp, future_timestamp) * fix doc typo * remove unecessary helper function design * fix deep model design bugs * add validation set support for rolling_window_dataset * simplify code implementation of rolling window dataset * fix dataset shuffle bug * fix rolling window test typo * change configuration names of specific deep models * fix rename typo * address various change requests and add documentation for deep models * Fix docs & torch requirement. * Make SARIMA more flexible. * Fix some bugs & simplify RollingWindowDataset. * Remove sklearn from ts_datasets deps. * Add warning suppression to ETS. * remove start_token argument for etsformer * Add warning suppression to ETS. * Improve M4 loader. * Remove legacy SARIMA import. * Minor fixes for DeepAR & ETSformer. * change implementation of deep ar to make sure it can predict arbitrary length of forecastings * change ambiguous function names * delete old lstm models Co-authored-by: Aadyot Bhatnagar <abhatnagar@salesforce.com>
salesforce · Jan 3, 2023 · a241a62 · a241a62
1 parent cc6e9a3
commit a241a62
Show file tree

Hide file tree

Showing 42 changed files with 3,694 additions and 638 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
 - repo: https://github.com/psf/black
-  rev: '19.3b0'
+  rev: '22.10.0'
   hooks:
   - id: black
     args: ["--line-length", "120"]

diff --git a/docs/source/merlion.models.anomaly.forecast_based.rst b/docs/source/merlion.models.anomaly.forecast_based.rst
@@ -12,7 +12,6 @@ anomaly.forecast\_based
     sarima
     ets
     prophet
-    lstm
     mses
 
 anomaly.forecast\_based.base
@@ -55,14 +54,6 @@ anomaly.forecast\_based.prophet
    :undoc-members:
    :show-inheritance:
 
-anomaly.forecast\_based.lstm
-----------------------------
-
-.. automodule:: merlion.models.anomaly.forecast_based.lstm
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 anomaly.forecast\_based.mses
 ----------------------------
 

diff --git a/docs/source/merlion.models.forecast.rst b/docs/source/merlion.models.forecast.rst
@@ -10,6 +10,7 @@ Base classes:
 
 .. autosummary::
     base
+    deep_base
     sklearn_base
 
 Univariate models:
@@ -20,13 +21,18 @@ Univariate models:
     ets
     prophet
     smoother
-    lstm
 
 `Multivariate <tutorials/forecast/2_ForecastMultivariate>` models:
 
 .. autosummary::
     vector_ar
     trees
+    deep_ar
+    autoformer
+    etsformer
+    informer
+    transformer
+
 
 `Exogenous regressor <tutorials/forecast/3_ForecastExogenous>` models:
 
@@ -37,6 +43,16 @@ Univariate models:
     vector_ar
     arima
 
+Deep Learning models:
+
+.. autosummary::
+    deep_ar
+    autoformer
+    etsformer
+    informer
+    transformer
+
+
 Note that the AutoML variants
 :py:mod:`AutoSarima <merlion.models.automl.autosarima>` and
 :py:mod:`AutoProphet <merlion.models.automl.autoprophet>`
@@ -53,6 +69,13 @@ forecast.base
    :undoc-members:
    :show-inheritance:
 
+forecast.deep\_base
+^^^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.deep_base
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 forecast.sklearn\_base
 ^^^^^^^^^^^^^^^^^^^^^^
 .. automodule:: merlion.models.forecast.sklearn_base
@@ -99,13 +122,6 @@ forecast.smoother
    :undoc-members:
    :show-inheritance:
 
-forecast.lstm
-^^^^^^^^^^^^^
-.. automodule:: merlion.models.forecast.lstm
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 Multivariate models
 -------------------
 
@@ -122,3 +138,43 @@ forecast.trees
    :members:
    :undoc-members:
    :show-inheritance:
+
+forecast.deep\_ar
+^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.deep_ar
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+forecast.autoformer
+^^^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.autoformer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+forecast.etsformer
+^^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.etsformer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+forecast.informer
+^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.informer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+forecast.transformer
+^^^^^^^^^^^^^^^^^^^^
+.. automodule:: merlion.models.forecast.transformer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+
+
diff --git a/docs/source/merlion.models.rst b/docs/source/merlion.models.rst
@@ -60,6 +60,7 @@ Finally, we support ensembles of models in :py:mod:`merlion.models.ensemble`.
     defaults
     factory
     base
+    deep_base
     layers
     anomaly
     anomaly.change_point
@@ -111,10 +112,19 @@ base
    :undoc-members:
    :show-inheritance:
 
+deep\_base
+----------
+
+.. automodule:: merlion.models.deep_base
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 layers
 ------
 
 .. automodule:: merlion.models.layers
    :members:
    :undoc-members:
    :show-inheritance:
+
diff --git a/docs/source/merlion.models.utils.rst b/docs/source/merlion.models.utils.rst
@@ -7,10 +7,20 @@ utils
    :show-inheritance:
 
 .. autosummary::
+    time_features
     rolling_window_dataset
+    early_stopping
     autosarima_utils
 
 
+utils.time\_features
+--------------------
+.. automodule:: merlion.models.utils.time_features
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 utils.rolling\_window\_dataset
 ------------------------------
 
@@ -20,11 +30,18 @@ utils.rolling\_window\_dataset
    :show-inheritance:
 
 
+utils.early\_stopping
+---------------------
+.. automodule:: merlion.models.utils.early_stopping
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 utils.autosarima\_utils
 -----------------------
 
 .. automodule:: merlion.models.utils.autosarima_utils
    :members:
    :undoc-members:
-   :show-inheritance:
-
+   :show-inheritance:
diff --git a/merlion/dashboard/models/anomaly.py b/merlion/dashboard/models/anomaly.py
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2022 salesforce.com, inc.
+# Copyright (c) 2023 salesforce.com, inc.
 # All rights reserved.
 # SPDX-License-Identifier: BSD-3-Clause
 # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause
@@ -25,7 +25,6 @@ class AnomalyModel(ModelMixin, DataMixin):
         "DynamicBaseline",
         "IsolationForest",
         "ETSDetector",
-        "LSTMDetector",
         "MSESDetector",
         "ProphetDetector",
         "RandomCutForest",

diff --git a/merlion/dashboard/models/forecast.py b/merlion/dashboard/models/forecast.py
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2022 salesforce.com, inc.
+# Copyright (c) 2023 salesforce.com, inc.
 # All rights reserved.
 # SPDX-License-Identifier: BSD-3-Clause
 # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause
@@ -28,7 +28,6 @@ class ForecastModel(ModelMixin, DataMixin):
         "Prophet",
         "AutoProphet",
         "Sarima",
-        "LSTM",
         "VectorAR",
         "RandomForestForecaster",
         "ExtraTreesForecaster",

diff --git a/merlion/models/anomaly/__init__.py b/merlion/models/anomaly/__init__.py
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2021 salesforce.com, inc.
+# Copyright (c) 2023 salesforce.com, inc.
 # All rights reserved.
 # SPDX-License-Identifier: BSD-3-Clause
 # For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/BSD-3-Clause
@@ -47,8 +47,8 @@
     -   trains the model on the time series ``train_data``
     -   ``anomaly_labels`` (optional): a time series aligned with ``train_data``, which indicates whether each
         time stamp is anomalous
-    -   ``train_config`` (optional): extra configuration describing how the model should be trained (e.g. learning rate
-        for the `LSTMDetector`). Not used for all models. Class-level default provided for models which do use it.
+    -   ``train_config`` (optional): extra configuration describing how the model should be trained. 
+        Not used for all models. Class-level default provided for models which do use it.
     -   ``post_rule_train_config``: extra configuration describing how to train the model's post-rule. Class-level
         default is provided for all models.
     -   returns a time series of anomaly scores produced by the model on ``train_data``.

diff --git a/merlion/models/anomaly/forecast_based/lstm.py b/merlion/models/anomaly/forecast_based/lstm.py