Add Prophet Regressor #2242

ParthivNaresh · 2021-05-09T00:14:07Z

Thanks to a conversation with @dsherry , @kmax12 , and @tyler3991, we made the decision to include Prophet alongside the cmdstanpy backend. The reasoning behind this is that because we aren't distributing Prophet code (simply requiring the user to install it as a dependency) we are in the same position as Prophet is regarding its use of pystan 2 and its GPLv3 license. As we aren't distributing the underlying code, we (and Prophet) aren't required to be under the same copyleft license.

However, because we have to think downstream to Tempo and whether it will be installed on-premises (which would require packaging up dependencies and distributing them), we can't rely on pystan 2 as a backend. Because Prophet doesn't support pystan 3 yet (which uses a permissive license), we will have to rely on cmdstanpy.

For more information read here

This PR only deals with adding the estimator so we can get it into main. I'll be raising another PR to add it to AutoML alongside perf tests.

codecov · 2021-05-09T03:06:57Z

Codecov Report

Merging #2242 (879c147) into main (0c64266) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2242     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        293     295      +2     
  Lines      26894   27055    +161     
=======================================
+ Hits       26851   27012    +161     
  Misses        43      43

Impacted Files	Coverage Δ
evalml/pipelines/__init__.py	`100.0% <ø> (ø)`
evalml/pipelines/components/__init__.py	`100.0% <ø> (ø)`
evalml/pipelines/components/estimators/__init__.py	`100.0% <ø> (ø)`
...alml/tests/model_family_tests/test_model_family.py	`100.0% <ø> (ø)`
evalml/tests/utils_tests/test_dependencies.py	`85.2% <ø> (ø)`
evalml/utils/gen_utils.py	`99.6% <ø> (ø)`
evalml/model_family/model_family.py	`100.0% <100.0%> (ø)`
...lines/components/estimators/regressors/__init__.py	`100.0% <100.0%> (ø)`
...ponents/estimators/regressors/prophet_regressor.py	`100.0% <100.0%> (ø)`
evalml/tests/component_tests/test_components.py	`100.0% <100.0%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c64266...879c147. Read the comment docs.

ParthivNaresh · 2021-05-17T16:23:32Z

A current issue I'm facing regarding the cmdstanpy backend is during following these instructions.

After installation of the backend cmdstanpy, a function has to be run from the installed package: install_cmdstan.py
This is responsible for downloading CmdStan from GitHub and building the CmdStan utilities. After this is done, a new directory is create in the default $HOME location (or whatever directory was specified when python install_cmdstan.py was run) by the name of cmdstan-2.26.1,

The path to this folder has to be assigned to the environment variable CMDSTAN. It can either be assigned to it separately as
export CMDSTAN=<path/to/dir>
or as part of the final installation command for Prophet through:
CMDSTAN=<path/to/dir> STAN_BACKEND=CMDSTANPY pip install prophet

Prophet includes pystan naturally as part of the installation process. To specify using cmdstanpy, a parameter has to be passed during Prophet instantiation: stan_backend='CMDSTANPY. If this parameter isn't specified, it will revert to the pystan implementation.

m = prophet.forecaster.Prophet()
print(f'Using backend: {m.stan_backend.get_type()}')
>>> Using backend: PYSTAN

If the backend is specified as cmdstanpy, an error is thrown:

m = prophet.forecaster.Prophet(stan_backend='CMDSTANPY')
print(f'Using backend: {m.stan_backend.get_type()}')
>>> ValueError: no such file /Users/parthiv.naresh/miniconda3/envs/prophet/lib/python3.8/site-packages/prophet/stan_model/prophet_model.bin

It appears to be looking for a file, prophet_model.bin which doesn't exist, whereas if the default pystan backend is used, then the file prophet_model.pkl is used, which does exist.

This has been filed with Prophet.

freddyaboulton

@ParthivNaresh This is looking good! I have some broad comments on the installation process and how we can make that easier for users. Component logic looks good but I'll also take a closer look later!

windows-requirements.txt

freddyaboulton · 2021-06-24T16:16:12Z

.github/workflows/linux_unit_tests_with_latest_deps.yml

@@ -66,11 +66,16 @@ jobs:
          ! pip freeze | grep -E "xgboost|catboost|lightgbm|plotly|ipywidgets|category_encoders"
          exit $?
      - if: ${{ !matrix.core_dependencies }}
-        name: Installing Dependencies
+        name: Installing Dependencies and Prophet


Right now we're installing our dependencies (which installs prophet since it's in our requirements file) and then we have to reinstall prophet with the right backend.

Although that works, I think we can simplify it a bit. I tried this out locally, and it seems to work (in the sense the unit tests pass)

pip install cmdstanpy==0.9.68 <compile cmdstan> export CMDSTAN=<path-to-cmdstan> export STAN_BACKEND=CMDSTANPY pip install -r dev-requirements.txt

I think this also makes it easier to communicate to users how to properly install prophet with the right backend.

This reminds me, we should add a section to the install page in the docs about prophet installation! I'm hoping it can be this:

pip install cmdstanpy==0.9.68 <compile cmdstan> export CMDSTAN=<path-to-cmdstan> export STAN_BACKEND=CMDSTANPY pip install evalml

Good call, hopefully we should be able to remove this once we can get the cmdstan-ext wheel up for an easy pip install

Update on this: we're not including Prophet in requirements because it's being installed as an extra-requirement in setup.py. Therefore we'll be including the entire installation process for Prophet for git-test-automl and git-test-other.

evalml/utils/gen_utils.py

freddyaboulton · 2021-06-24T16:21:05Z

evalml/pipelines/components/estimators/regressors/prophet_regressor.py

+        holidays_prior_scale=10,
+        seasonality_mode="additive",
+        random_seed=0,
+        stan_backend="CMDSTANPY",


This isn't meant to be changed right? I thought the plan was to only support prophet with the CMDSTANPY backend?

So we need this to be specified so we can unit test it. Unfortunately the parameter doesn't get included in the attributes list unless it's passed into the Prophet component

evalml/pipelines/components/estimators/regressors/prophet_regressor.py

evalml/tests/component_tests/test_prophet_regressor.py

ParthivNaresh · 2021-08-02T12:17:43Z

.github/workflows/linux_unit_tests_with_latest_deps.yml

+          make installdeps-test
+          pip uninstall pystan -y
+          pip freeze
+      - if: ${{ !matrix.core_dependencies && (matrix.command == 'git-test-modelunderstanding' || matrix.command == 'git-test-dask') }}


Don't install prophet for non-core dependencies and if the tests are git-test-modelunderstanding and git-test-dask because they don't need it

.github/workflows/linux_unit_tests_with_minimum_deps.yml

chukarsten

Nice work, Parthiv! I think that addressing @freddyaboulton 's concerns about the tests taking 15s each is very important to not bloating the CI time. If you can reduce those and ping me with the results, I'd love to follow up. Additionally, with respect to the suppression of stdout, I went through the ref and it seemed like some people had issues crop up with "running out of file descriptors" or something of the like. For the sake of getting the Prophet regressor in, I'm willing to accept the risk of some weirdness here, but I'd like an issue filed to try and address this differently post-merge as I think this has the potential of being tech debt with a high interest rate, ala the weirdness going on with the matplotlib pngs.

.github/workflows/linux_unit_tests_with_latest_deps.yml

evalml/pipelines/components/estimators/regressors/prophet_regressor.py

ParthivNaresh · 2021-08-03T11:11:22Z

setup.py

 from setuptools import find_packages, setup

 with open("README.md", "r") as fh:
    long_description = fh.read()

 extras_require = {
    'update_checker': ['alteryx-open-src-update-checker >= 2.0.0'],
+    'prophet': ['cmdstan-builder == 0.0.3']


This links to the cmdstan-builder pip package.
The link for it on GitHub is here, and the PyPI package is here.
To learn more about the thought process behind this, please take a look at this, this, this, this, this and this.

setup.py

freddyaboulton

Solid work @ParthivNaresh ! I think this looks great. I mainly have some questions about the install process that I want to resolve before merge.

README.md

.github/workflows/linux_unit_tests_with_minimum_deps.yml

evalml/tests/component_tests/test_prophet_regressor.py

setup.py

evalml/utils/gen_utils.py

evalml/tests/utils_tests/test_dependencies.py

evalml/pipelines/components/estimators/regressors/prophet_regressor.py

freddyaboulton · 2021-08-03T15:53:49Z

evalml/tests/component_tests/test_components.py

@@ -1297,6 +1312,8 @@ def test_estimators_accept_all_kwargs(
        )
    if estimator_class.model_family == ModelFamily.ENSEMBLE:
        params = estimator.parameters
+    elif estimator_class.model_family == ModelFamily.PROPHET:
+        params = estimator.get_params()


Would using estimator.parameters work for prophet?

Yes that would work. I'm testing this in particular because Prophet has its own dictionary representation that covers a wide range of parameters being passed into the component.

bchen1116

This looks great! Thanks for explaining how things work in the writeups/comments/calls! Glad we're able to add this to EvalML!

ParthivNaresh self-assigned this May 17, 2021

ParthivNaresh force-pushed the Add-Prophet-Regressor branch from 0b57023 to d348666 Compare June 8, 2021 16:26

ParthivNaresh marked this pull request as ready for review June 23, 2021 17:43

ParthivNaresh requested review from angela97lin, dsherry, freddyaboulton, bchen1116, christopherbunn, chukarsten and jeremyliweishih and removed request for dsherry June 23, 2021 17:49

freddyaboulton reviewed Jun 24, 2021

View reviewed changes

ParthivNaresh commented Aug 2, 2021

View reviewed changes

.github/workflows/linux_unit_tests_with_minimum_deps.yml Show resolved Hide resolved

chukarsten approved these changes Aug 2, 2021

View reviewed changes

.github/workflows/linux_unit_tests_with_latest_deps.yml Show resolved Hide resolved

.github/workflows/linux_unit_tests_with_latest_deps.yml Outdated Show resolved Hide resolved

evalml/pipelines/components/estimators/regressors/prophet_regressor.py Outdated Show resolved Hide resolved

ParthivNaresh commented Aug 3, 2021

View reviewed changes

setup.py Outdated Show resolved Hide resolved

freddyaboulton approved these changes Aug 3, 2021

View reviewed changes

bchen1116 approved these changes Aug 3, 2021

View reviewed changes

final commit

9ec1cf4

ParthivNaresh force-pushed the Add-Prophet-Regressor branch from 628fb6e to 9ec1cf4 Compare August 3, 2021 19:18

ParthivNaresh added 4 commits August 3, 2021 16:25

remove pystan uninstallation from CI

aa16b37

same as before

1d5b7c3

Merge branch 'main' into Add-Prophet-Regressor

c4af9dc

dependency update

879c147

ParthivNaresh merged commit 455210b into main Aug 4, 2021

freddyaboulton mentioned this pull request Aug 4, 2021

Prophet: speed up CI #2588

Closed

chukarsten mentioned this pull request Aug 12, 2021

Release v0.30.1 #2623

Closed

freddyaboulton deleted the Add-Prophet-Regressor branch May 13, 2022 15:34

Add Prophet Regressor #2242

Add Prophet Regressor #2242

ParthivNaresh commented May 9, 2021 •

edited

codecov bot commented May 9, 2021 •

edited

ParthivNaresh commented May 17, 2021 •

edited

freddyaboulton left a comment

freddyaboulton Jun 24, 2021

ParthivNaresh Jul 8, 2021

ParthivNaresh Aug 2, 2021

freddyaboulton Jun 24, 2021

ParthivNaresh Jul 30, 2021

ParthivNaresh Aug 2, 2021

chukarsten left a comment

ParthivNaresh Aug 3, 2021

freddyaboulton left a comment

freddyaboulton Aug 3, 2021

ParthivNaresh Aug 3, 2021 •

edited

bchen1116 left a comment

Add Prophet Regressor #2242

Add Prophet Regressor #2242

Conversation

ParthivNaresh commented May 9, 2021 • edited

codecov bot commented May 9, 2021 • edited

Codecov Report

ParthivNaresh commented May 17, 2021 • edited

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ParthivNaresh Aug 3, 2021 • edited

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

ParthivNaresh commented May 9, 2021 •

edited

codecov bot commented May 9, 2021 •

edited

ParthivNaresh commented May 17, 2021 •

edited

ParthivNaresh Aug 3, 2021 •

edited