Horovod cannot be installed together with ML frameworks with pip>=22.0

[EnricoMi]

Since pip>=22.0, the following way to install Horovod does not work any more:

HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]

Even though horovod[tensorflow] depends on tensorflow, pip does not install `tensorflow before attempting to install horovod.

This probably holds for any ML framework, not only tensorflow.

With latest PIP:

$ python -m pip install --upgrade --force pip
...
Successfully installed pip-22.0.4

$ HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]
Collecting tensorflow==2.7.0
  Using cached tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (489.6 MB)
Collecting horovod[tensorflow]
  Using cached horovod-0.24.2.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... done
...
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... error
  ...
      ModuleNotFoundError: No module named 'tensorflow'
      CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
        Could NOT find Tensorflow (missing: Tensorflow_LIBRARIES) (Required is at
        least version "1.15.0")
  ....
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: horovod, tensorflow
  Running setup.py install for horovod ... error
...

Looks like an older PIP swaps the order of installing horovod and tensorflow when horovod fails, continues to install tensorflow and attempts horovod again, which then works fine:

$ python -m pip install --upgrade --force "pip<22.0"
...
Successfully installed pip-21.3.1

$ HOROVOD_WITH_TENSORFLOW=1 pip install tensorflow==2.7.0 horovod[tensorflow]
Collecting tensorflow==2.7.0
  Using cached tensorflow-2.7.0-cp38-cp38-manylinux2010_x86_64.whl (489.6 MB)
Collecting horovod[tensorflow]
  Using cached horovod-0.24.2.tar.gz (3.4 MB)
  Preparing metadata (setup.py) ... done
...
Building wheels for collected packages: horovod
  Building wheel for horovod (setup.py) ... error
...
  ERROR: Failed building wheel for horovod
  Running setup.py clean for horovod
Failed to build horovod
Installing collected packages: tensorflow, horovod
    Running setup.py install for horovod ... done
Successfully installed horovod-0.24.2 tensorflow-2.7.0

See

Installing collected packages: horovod, tensorflow

vs.

Installing collected packages: tensorflow, horovod

after

ERROR: Failed building wheel for horovod

[maxhgerlach]

I think a pyproject.toml would be the right place these days to specify build-time requirements. https://packaging.python.org/en/latest/specifications/declaring-build-dependencies/
https://setuptools.pypa.io/en/latest/userguide/quickstart.html#basic-use

However, I'm not sure if such a file supports having different build-time requirements for different "extras" like [tensorflow] or [pytorch].

[maxhgerlach]

Oof, here's a related discussion concerning a different package: pypa/pipenv#4482 (comment)

Apparently, with pyproject.toml build-time requirements, one is supposed to manage separate packages horovod-tensorflow, horovod-pytorch, horovod-allframeworks, etc.

[maxhgerlach]

One way to go could be to stick with setuptools to define these requirements, but to alter the setup_requires depending on which environment variables are set (currently we only alter extras_require). So HOROVOD_WITH_TENSORFLOW=1 pip install horovod would depend on TensorFlow then to run setup.py.

h5p still handles an optional build-time dependency on mpi4py this way: https://github.com/h5py/h5py/blob/3051c76afa73ad0c72d2c7c6724fe3617d2fefd8/setup.py#L45

Related discussion initiated by an author of h5py: https://discuss.python.org/t/options-to-build-the-same-package-in-different-ways-with-different-build-dependencies/4458/16

[EnricoMi]

The simplest way is to install the ML frameworks in a separate step first (as we do it in our Dockerfile.test.?pu), and then run pip install horovod[FRAMEWORK] to get framework-specific dependencies.