Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No module named 'packaging' when install horovod #3744

Closed
ice-tong opened this issue Oct 14, 2022 · 10 comments · Fixed by #3745
Closed

No module named 'packaging' when install horovod #3744

ice-tong opened this issue Oct 14, 2022 · 10 comments · Fixed by #3745
Labels
bug

Comments

@ice-tong
Copy link

ice-tong commented Oct 14, 2022 •
edited

It seems that the horovod v0.26.0 has some dependency problems.

How long does it take for a new patch version to be released or should I pin the horovod version? ^_^

image
@ice-tong ice-tong added the bug label Oct 14, 2022
@ice-tong
Copy link
Author

ping @tgaddair have a look ~

@EnricoMi
Copy link
Collaborator

EnricoMi commented Oct 14, 2022 •
edited

Hotfix release is on its way (today): #3745 #3741

@EnricoMi EnricoMi mentioned this issue Oct 14, 2022
Merged
@ice-tong
Copy link
Author

Hotfix release is on its way (today): #3745 #3741

Thanks for the reply!

@EnricoMi
Copy link
Collaborator

EnricoMi commented Oct 14, 2022 •
edited

Fix has been released.

@EnricoMi EnricoMi mentioned this issue Oct 27, 2022
Open
@itsdani
Copy link

itsdani commented Oct 27, 2022

This issue isn't fully fixed, install still fails with pip install --use-pep517 --no-cache-dir horovod==0.26.1 and when trying to install with poetry 1.2.2

The fix moved the from packaging import version into get_cmake_bin, but there it can still fail:

        File "/tmp/pip-build-env-yyt1ftmx/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "<string>", line 106, in build_extensions
        File "<string>", line 67, in get_cmake_bin
      ModuleNotFoundError: No module named 'packaging'
      [end of output]

This is the full output of pip:

Details 
 ▲ ~/work/horovodtest pip install --use-pep517 --no-cache-dir horovod==0.26.1                                                                                                              10s
Collecting horovod==0.26.1
  Downloading horovod-0.26.1.tar.gz (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 8.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyyaml in ./.venv/lib/python3.10/site-packages (from horovod==0.26.1) (6.0)
Requirement already satisfied: psutil in ./.venv/lib/python3.10/site-packages (from horovod==0.26.1) (5.9.3)
Requirement already satisfied: packaging in ./.venv/lib/python3.10/site-packages (from horovod==0.26.1) (21.3)
Requirement already satisfied: cffi>=1.4.0 in ./.venv/lib/python3.10/site-packages (from horovod==0.26.1) (1.15.1)
Requirement already satisfied: cloudpickle in ./.venv/lib/python3.10/site-packages (from horovod==0.26.1) (2.2.0)
Requirement already satisfied: pycparser in ./.venv/lib/python3.10/site-packages (from cffi>=1.4.0->horovod==0.26.1) (2.21)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.10/site-packages (from packaging->horovod==0.26.1) (3.0.9)
Building wheels for collected packages: horovod
  Building wheel for horovod (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for horovod (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [236 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/horovod
      copying horovod/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod
      creating build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/sync_batch_norm.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/optimizer.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/functions.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/compression.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      copying horovod/torch/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/torch
      creating build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/sync_batch_norm.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation_eager.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/functions.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow
      creating build/lib.linux-x86_64-cpython-310/horovod/spark
      copying horovod/spark/runner.py -> build/lib.linux-x86_64-cpython-310/horovod/spark
      copying horovod/spark/mpi_run.py -> build/lib.linux-x86_64-cpython-310/horovod/spark
      copying horovod/spark/gloo_run.py -> build/lib.linux-x86_64-cpython-310/horovod/spark
      copying horovod/spark/conf.py -> build/lib.linux-x86_64-cpython-310/horovod/spark
      copying horovod/spark/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark
      creating build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/task_fn.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/run_task.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/mpi_run.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/launch.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/js_run.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/gloo_run.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      copying horovod/runner/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner
      creating build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/worker.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/utils.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/strategy.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/runner.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/ray_logger.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/elastic_v2.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/driver_service.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/adapter.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      copying horovod/ray/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/ray
      creating build/lib.linux-x86_64-cpython-310/horovod/mxnet
      copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-cpython-310/horovod/mxnet
      copying horovod/mxnet/functions.py -> build/lib.linux-x86_64-cpython-310/horovod/mxnet
      copying horovod/mxnet/compression.py -> build/lib.linux-x86_64-cpython-310/horovod/mxnet
      copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/mxnet
      creating build/lib.linux-x86_64-cpython-310/horovod/keras
      copying horovod/keras/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/keras
      copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-cpython-310/horovod/keras
      copying horovod/keras/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/keras
      creating build/lib.linux-x86_64-cpython-310/horovod/data
      copying horovod/data/data_loader_base.py -> build/lib.linux-x86_64-cpython-310/horovod/data
      copying horovod/data/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/data
      creating build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/util.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/process_sets.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/exceptions.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/basics.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      copying horovod/common/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/common
      creating build/lib.linux-x86_64-cpython-310/horovod/_keras
      copying horovod/_keras/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/_keras
      copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-cpython-310/horovod/_keras
      copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/_keras
      creating build/lib.linux-x86_64-cpython-310/horovod/torch/elastic
      copying horovod/torch/elastic/state.py -> build/lib.linux-x86_64-cpython-310/horovod/torch/elastic
      copying horovod/torch/elastic/sampler.py -> build/lib.linux-x86_64-cpython-310/horovod/torch/elastic
      copying horovod/torch/elastic/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/torch/elastic
      creating build/lib.linux-x86_64-cpython-310/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/keras
      creating build/lib.linux-x86_64-cpython-310/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_worker.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_service.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/data
      copying horovod/tensorflow/data/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/tensorflow/data
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/torch
      copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/torch
      copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/torch
      copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/torch
      copying horovod/spark/torch/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/torch
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/compute_worker.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/tensorflow
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/task
      copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/task
      copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/task
      copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/task
      copying horovod/spark/task/gloo_exec_fn.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/task
      copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/task
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/util.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/remote.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/legacy.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/estimator.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/datamodule.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      copying horovod/spark/lightning/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/lightning
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/datamodule.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      copying horovod/spark/keras/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/keras
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/rsh.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/rendezvous.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/host_discovery.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/driver
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/pytorch_data_loaders.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/data_loaders
      creating build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/util.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/store.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/params.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/datamodule.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      copying horovod/spark/common/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/spark/common
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/threads.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/streams.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/remote.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/network.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/lsf.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/cache.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      copying horovod/runner/util/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/util
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/task
      copying horovod/runner/task/task_service.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/task
      copying horovod/runner/task/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/task
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/http
      copying horovod/runner/http/http_server.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/http
      copying horovod/runner/http/http_client.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/http
      copying horovod/runner/http/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/http
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/worker.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/settings.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/rendezvous.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/registration.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/driver.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/discovery.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/constants.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      copying horovod/runner/elastic/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/elastic
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/driver
      copying horovod/runner/driver/driver_service.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/driver
      copying horovod/runner/driver/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/driver
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/common
      copying horovod/runner/common/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/tiny_shell_exec.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/timeout.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/settings.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/secret.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/network.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/hosts.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/host_hash.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/env.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/config_parser.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/codec.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      copying horovod/runner/common/util/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/util
      creating build/lib.linux-x86_64-cpython-310/horovod/runner/common/service
      copying horovod/runner/common/service/task_service.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/service
      copying horovod/runner/common/service/driver_service.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/service
      copying horovod/runner/common/service/compute_service.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/service
      copying horovod/runner/common/service/__init__.py -> build/lib.linux-x86_64-cpython-310/horovod/runner/common/service
      running build_ext
      Traceback (most recent call last):
        File "/home/dsegesdi/work/horovodtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/dsegesdi/work/horovodtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/dsegesdi/work/horovodtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 412, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 483, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 213, in <module>
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
          self.run_command('build')
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-ma4zyx42/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
          self.build_extensions()
        File "<string>", line 106, in build_extensions
        File "<string>", line 67, in get_cmake_bin
      ModuleNotFoundError: No module named 'packaging'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for horovod
Failed to build horovod
ERROR: Could not build wheels for horovod, which is required to install pyproject.toml-based projects

@itsdani itsdani mentioned this issue Oct 27, 2022
Closed
4 tasks
@GitHubChuanYu
Copy link

Hi itsdani,

Do you have the ModuleNotFoundError: No module named 'packaging' issue in get_cmake_in fixed? I got the same error while pip install --use-pep517 horovod==0.27.0.

@itsdani
Copy link

itsdani commented Jun 4, 2023 •
edited

No, horovod is not compatible with pep517 (and poetry, since it forces pep517), which enforces isolated build environments. Horovod doesn't specify build-dependencies explicitly in a pyproject.toml (like packaging and setuptools), so they aren't present in the isolated build environment, which is why the ModuleNotFoundError happens.

But the solution is not as simple as adding these to a pyproject.toml as dependencies, because horovod also has a lot of other optional dependencies (e.g. tensorflow) and these should be dynamically added to the build environment, probably by using a custom build-backend script. I have tried making the POC for such a backend in #3758 but I'm not familiar enough with the horovod build to write an actually useable one. So at this point I gave up on making this work.

There is some more discussion about this on PR #3758 and issue #3697

As far as I know this hasn't been dealt with since then, but I don't really follow the topic anymore so I might be wrong

@GitHubChuanYu
Copy link

Hi Daniel,

Thanks for your reply. Just a general question, if horovod is not compatible with pep517 right now, is there any other way to install horovod successfully right now, I need to use it for distributed training?

Thanks!

@itsdani
Copy link

itsdani commented Jun 5, 2023

As a hacky workaround we removed horovod as a project dependency, activated our venv (managed by poetry) and installed horovod with pip manually, without using pep517 isolation. We used something like this in our install script:

poetry install
poetry run pip install horovod

@maxhgerlach
Copy link
Collaborator

I think currently the typical way to go would be to have a separate build step for Horovod. For example: Set up a virtual environment with your version of TensorFlow, PyTorch, and/or MXNet. Inside this virtual environment you could build a binary wheel of Horovod that fits your environment and tag it with a local version identifier, e.g.: HOROVOD_LOCAL_VERSION=your.site.0 pip wheel -v --no-deps horovod==0.28.1. Store the resulting wheel somewhere (shared directory, local package repository, ...). In poetry configure that location as a source and add horovod as a dependency with a version number like 0.28.1+your.site.0`.

@Borda Borda mentioned this issue Jul 17, 2023
Merged
3 tasks
@flixxox flixxox mentioned this issue Nov 10, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bum version to 0.26.1 horovod/horovod
5 participants
@maxhgerlach @itsdani @ice-tong @GitHubChuanYu @EnricoMi