Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make prophet, lightgbm and catboost optional #1589

Merged
merged 14 commits into from
Aug 3, 2023
Merged

Conversation

hrzn
Copy link
Contributor

@hrzn hrzn commented Feb 22, 2023

Now that I have access to an M1 Mac, I was able to run more tests.

This is up for discussion, but I propose the following:

  • We make prophet optional again, and not included anymore in the darts package. That's because it's just too painful to install and I'm not even sure it can always be done on Apple silicon without using x64 emulation, which is not satisfying. Having it as a hard dependency is too much troubles for many users I suspect...
  • Adapt the INSTALL guide to simplify things for LightGBM and Apple Silicon, and add a section explaining that now prophet has to be installed separately.

Edit (@dennisbader, 2023-08-03):
Removed Prophet, LightGBM, and CatBoost dependencies from PyPI packages (darts, u8darts, u8darts[torch]), and conda-forge packages (u8darts, u8darts-torch) to avoid installation issues that some users were facing (installation on Apple M1/M2 devices, ...).

  • The models are still supported by installing the required packages as described in our installation guide.
  • PyPi package u8darts[all] and conda-forge package u8darts-all are now equivalent to the old darts package (all dependencies).
  • Added new PyPI flavor u8darts[notorch], and conda-forge flavor u8darts-notorch which are equivalent to the old u8darts installation.

@Eliotdoesprogramming
Copy link
Contributor

@hrzn

Just wanted to comment to add my own user experience on a M1 Macbook (2020 macbook pro) if its all helpful to you:

I have no issues installing prophet on my m1, even on arm64 architecture. I did however have issues with LightGBM installation through pip, but was easily fixed with an install through conda forge (all conda environments in this example, not virtualenv)

I did have issues with installing prophet on windows x86_64 architecture, but solved by using WSL.

@hrzn
Copy link
Contributor Author

hrzn commented Feb 23, 2023

@hrzn

Just wanted to comment to add my own user experience on a M1 Macbook (2020 macbook pro) if its all helpful to you:

I have no issues installing prophet on my m1, even on arm64 architecture. I did however have issues with LightGBM installation through pip, but was easily fixed with an install through conda forge (all conda environments in this example, not virtualenv)

I did have issues with installing prophet on windows x86_64 architecture, but solved by using WSL.

Thanks for sharing. I'm now wondering if I did anything wrong when trying to install Prophet. I tried a couple of ways, including different versions of Python, installing pystan or cmdstan first etc. How are you installing it?

For the record, here's what I get when I attempt pip install prophet:

      --- CmdStan v2.26.1 built ---
      Test model compilation
      cmd: make examples/bernoulli/bernoulli
      cwd: None
      09:25:56 - cmdstanpy - WARNING - CmdStan installation failed.
      Command "make clean-all" failed
      Command ['make', 'examples/bernoulli/bernoulli']
          error during processing No such file or directory
      CmdStan installation failed.
      Command "make clean-all" failed
      Command ['make', 'examples/bernoulli/bernoulli']
          error during processing No such file or directory
      clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS         -c -MT stan/src/stan/model/model_header.hpp.gch -MT stan/src/stan/model/model_header.d -MM -E -MG -MP -MF stan/src/stan/model/model_header.d stan/src/stan/model/model_header.hpp
      clang++ -std=c++1y -Wno-unknown-warning-option -Wno-tautological-compare -Wno-sign-compare -D_REENTRANT -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2019_U8/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.72.0 -I stan/lib/stan_math/lib/sundials_5.6.1/include    -DBOOST_DISABLE_ASSERTS         -c -MT src/cmdstan/main.o -MM -E -MG -MP -MF src/cmdstan/main.d src/cmdstan/main.cpp

      --- Translating Stan model to C++ code ---
      bin/stanc  --o=examples/bernoulli/bernoulli.hpp examples/bernoulli/bernoulli.stan
      make: bin/stanc: Bad CPU type in executable
      make: *** [examples/bernoulli/bernoulli.hpp] Error 1
      Traceback (most recent call last):
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/julien/miniconda3/envs/darts/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 413, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 398, in _build_with_temp_dir
          self.run_setup()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 183, in <module>
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/private/var/folders/c3/0hfh1r6j2t944n9kwd8mg5c00000gn/T/pip-build-env-mr7_c1ya/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 144, in run
        File "<string>", line 131, in build_models
        File "<string>", line 110, in build_cmdstan_model
        File "<string>", line 94, in install_cmdstan_deps
      RuntimeError: CmdStan failed to install in repackaged directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for prophet
Failed to build prophet
ERROR: Could not build wheels for prophet, which is required to install pyproject.toml-based projects

@Eliotdoesprogramming
Copy link
Contributor

Eliotdoesprogramming commented Feb 24, 2023

honestly its just a normal pip install for me. here was my process:

conda create -n install_demo python=3.9 -c conda-forge
Screen Shot 2023-02-24 at 1 05 59 PM

pip install prophet --no-cache-dir # no cache for clean install
Screen Shot 2023-02-24 at 1 07 42 PM

brew list

Screen Shot 2023-02-24 at 1 09 14 PM

the one thing I can think of off the top of my head is I've installed mamba to make some of the package management faster
hope this helps!

@martinb-ai
Copy link

martinb-ai commented Feb 28, 2023

@hrzn Could we make lightgbm optional when installing darts pytorch? It is a pain to deal with on Macs 👎

Seems it always bundles together even if I don't use the lib.

@eshwen
Copy link

eshwen commented Mar 7, 2023

I've been struggling with this recently. But I think I've found a way to install both LightGBM and prophet without resorting to emulation.

As @Eliotdoesprogramming suggested, install LightGBM with conda:

conda install -c conda-forge 'lightgbm>=3.3.3'

As for prophet, I installed CmdStan through conda, then installed the Python client and rebuilt it:

conda install -c conda-forge cmdstan
pip install cmdstanpy
python -c "from cmdstanpy import rebuild_cmdstan; rebuild_cmdstan()"
export PROPHET_REPACKAGE_CMDSTAN=False

Then I was able to just pip install darts. My full list of instructions can be found here, and I'm happy to open a PR to fold them into INSTALL.md

@madtoinou
Copy link
Collaborator

Hi @eshwen,

Sorry for the delay and thank you for sharing your solution!

Could you please check if your approach still works after the release of PyTorch 2.0? If, a PR adding these instructions to the documentation would be very welcome!

@eshwen
Copy link

eshwen commented Apr 3, 2023

Hi @eshwen,

Sorry for the delay and thank you for sharing your solution!

Could you please check if your approach still works after the release of PyTorch 2.0? If, a PR adding these instructions to the documentation would be very welcome!

No problem. I've briefly tested, and it does still work with PyTorch 2.0. In fact, it doesn't appear to need the cmdstanpy block anymore. When I have some free time, I'll open a PR to update the documentation

@TueLindhart
Copy link

Not sure if this is the correct forum, but I found a way to install prophet on MacOS 13.2.1 with M2 processor.

  1. pip install cmdstanpy
  2. Use cmdstanpy to install CMDStan either by python or terminal. (See here)
  3. In terminal set export PROPHET_REPACKAGE_CMDSTAN=False. It is the repackaging which makes it fail.
  4. pip install prophet

I do not know if prophet actually works properly as I am not interesting in using it. However, I am forced to install prophet=1.1.1 to be able to use the newest darts features.

@ale-dg
Copy link

ale-dg commented May 15, 2023

Hi,

As a fellow M1 Mac user, I have found that most of the packages used in Darts fail with "used" or "old" environments (at least Conda environments). Therefore I'd suggest to create a new environment and install all the dependencies first before installing darts. It goes even smoother.

As for the specific case of Prophet, I have raised a couple of issues in their repo because the cmdstan shouldn't try to build the wheel locally since the pip command should be retrieving the built wheel. You can see it here facebook/prophet#2402. They have found the issue on Windows but still missing the solution for macOS.

On a second note, as for now, be sure to install prophet's version 1.1.1 since the newest one has an issue when trying to fit some data. Hence the suggestion to first install first Darts' external dependencies (i.e. PyTorch, Prophet, LightGBM, etc) and then Darts in whatever package you want.

Best

@dennisbader
Copy link
Collaborator

I will pick this up and additionally exclude lightgbm (there are issues for many users installing on M1) and catboost (catboost>=1.2.0 installation requires a lot of additional build tools (cmake, ninja, ...) which we should avoid forcing on the user)

@dennisbader dennisbader changed the title make prophet optional again, adapt install readme make prophet, lightgbm and catboost optional Aug 2, 2023
@codecov-commenter
Copy link

codecov-commenter commented Aug 2, 2023

Codecov Report

Patch coverage: 38.09% and project coverage change: +0.16% 🎉

Comparison is base (137cf8c) 93.72% compared to head (e0cd981) 93.88%.

❗ Current head e0cd981 differs from pull request most recent head eebd227. Consider uploading reports for the commit eebd227 to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1589      +/-   ##
==========================================
+ Coverage   93.72%   93.88%   +0.16%     
==========================================
  Files         131      132       +1     
  Lines       12662    12639      -23     
==========================================
- Hits        11867    11866       -1     
+ Misses        795      773      -22     
Files Changed Coverage Δ
darts/models/__init__.py 73.01% <10.00%> (+15.32%) ⬆️
darts/models/utils.py 60.00% <60.00%> (ø)
darts/models/forecasting/regression_model.py 97.02% <100.00%> (+1.78%) ⬆️

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

darts/models/utils.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@madtoinou madtoinou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the way the optional dependencies are handled, just spotted a small typo.

Copy link
Collaborator

@madtoinou madtoinou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

With this, XGBoost becomes the most extensively tested model of the library if the optional dependencies are not installed...

INSTALL.md Outdated
* Install darts with all available models (recommended): `conda install -c conda-forge -c pytorch u8darts-all`.
* Install darts with all available models: `conda install -c conda-forge -c pytorch u8darts-all`
* Install core only (without neural networks, Prophet, LightGBM and Catboost): `conda install -c conda-forge u8darts`
* Install core + Prophet + LightGBM + CatBoost: `pip install "u8darts[notorch]"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be coherent with the others, should this flavor of the package be called udarts-notorch?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

INSTALL.md Outdated
softwareupdate --install-rosetta
```

Below are the necessary instructions to create and configure the environment:
- Start by installing conda (e.g., with miniforge : `brew install miniforge`).
- Install conda if you haven't done so (e.g., with miniforge : `brew install miniforge`).
- Create the x_64 environment : `CONDA_SUBDIR=osx-64 conda create -n env_name python=3.9 pip`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update this to 3.10 to be coherent with the instruction above?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@dennisbader dennisbader merged commit ecacd93 into master Aug 3, 2023
9 checks passed
@dennisbader dennisbader deleted the feat/improve-install branch August 3, 2023 11:33
@dennisbader dennisbader mentioned this pull request Aug 1, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

9 participants