Install Dependencies throwing subprocess error

[hipsterette]

`Traceback (most recent call last):
File "mlops_install", line 5, in
File "/Users/*****/Documents/MLOPs/scripts/python/mlops_utils.py", line 126, in install_mlops_dependencies
pip_install(hou.text.expandString("$MLOPS/requirements.txt"), True, True)
File "/Users/****/Documents/MLOPs/scripts/python/mlops_utils.py", line 364, in pip_install
raise hou.Error(res[1].decode())
hou.Error: error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
/Applications/Houdini/Houdini19.5.752/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/init.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the distutils module in sys.modules. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
warnings.warn(
/Applications/Houdini/Houdini19.5.752/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
File "", line 2, in

    File "<pip-setuptools-caller>", line 34, in <module>
    File "/private/var/folders/6n/9xz_rjz946b80_8s2gw2jwkw0000gn/T/pip-install-k2hyvmzz/xformers_67ce5be12fa24c4786a46aadfad7bd4d/setup.py", line 23, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

    File "<pip-setuptools-caller>", line 34, in <module>
    File "/private/var/folders/6n/9xz_rjz946b80_8s2gw2jwkw0000gn/T/pip-install-k2hyvmzz/xformers_67ce5be12fa24c4786a46aadfad7bd4d/setup.py", line 23, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
`

[Huronsen]

I have the same issue. Have you already found a way to resolve it?

[kapkoy]

Same here

Traceback (most recent call last):
File "mlops_install", line 5, in
File "X:\AI/Stable Diffusion/MLOPs/scripts/python\mlops_utils.py", line 126, in install_mlops_dependencies
pip_install(hou.text.expandString("$MLOPS/requirements.txt"), True, True)
File "X:\AI/Stable Diffusion/MLOPs/scripts/python\mlops_utils.py", line 364, in pip_install
raise hou.Error(res[1].decode())
hou.Error: C:\PROGRA1\SIDEEF1\HOUDIN~1.506\python310\lib\site-packages_distutils_hack_init_.py:30: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Running command git clone --filter=blob:none --quiet https://github.com/Keith-Hon/bitsandbytes-windows.git 'C:\Users\thoma\AppData\Local\Temp\pip-install-gu84u8dx\bitsandbytes_d98542e03bbf40c4819db6ec8b356529'
error: subprocess-exited-with-error

[codebot6000]

just popping in to say also receiving "setuptools is replacing distutils with houdini 20.0.506 on linux

[FuzzyCactusLogic]

Same error here. Houdini 20.0.506 on Linux Mint

[gzimmermann123]

Same error on clean 20.0.547 install on macOS

[gzimmermann123]

I found a solution for the install dependencies: Don't use the shelf tool but do it manually via pip install. Then fix all the problems that you see. Here are the details:

I went ahead and tried installing the requirements via pip install in the shell (instead of the shelf tool provided). I found 2 problems when running:

pip install -r requirements.txt

1. pip install rembg[gpu] failed for me on a Mac (M1). I needed to install the CPU version instead:
   pip install rembg
   or modify the requirements.txt and replace "rembg[gpu]" with "rembg"

2. halpecocotools would not build out of the box. I needed to do the following:
   according to:
   #x86_64-linux-gnu-gcc: error: ../common/maskApi.c: No such file or directory (solution) HaoyiZhu/HalpeCOCOAPI#1

$ git clone https://github.com/HaoyiZhu/HalpeCOCOAPI.git
$ cd HalpeCOCOAPI/PythonAPI

edit halpecocotools/_mask.pyx:
replace # distutils: sources = ../common/maskApi.c with # distutils: sources = common/maskApi.c
according to this: #HaoyiZhu/HalpeCOCOAPI@a603f16
thank you @Ambrosiussen !

then finally
$ python3 setup.py build develop

now remove halpecocotools from the requirements.txt (since we just built it from source!)
after that I was able to run all requirements again and all went well:

pip install -r requirements.txt

Now that the dependencies have been setup, I tried Houdini:
Starting Houdini and making sure all these installed packages are in the PYTHONPATH I was now able to use the "Download Model" shelf tool successfully.

After that I was able to start playing with the MLOPs nodes!

[giosampietro]

@gzimmermann123 thanks for finding a solution. I’m a bit lost when it comes to implementing the steps you described. You mention “ I went ahead and tried installing the requirements via pip install in the shell ” do you mean the internal houdini python shell? Would you be ok with posting a few extra steps/descruptions to make this solution (finally!) available also to pyhon noobs?

@Ambrosiussen could the rembg[gpu] choice step be automated by the pip install process itself?

[gzimmermann123]

@giosampietro sorry my steps are not very detailed I am afraid.

The pip commands I mentioned (and the building of halpecocotools) need to be done in a shell outside of Houdini. Not a python shell but a regular system shell, bash or something. It doesn't really have anything to do with houdini. It is about installing the required python packages to make MLOPs happy. Are you on a Mac? If so the shell you should probably use is the one sidefx provides, see my screen shot. You can also use a regular Mac OS shell but this one has houdini stuff mounted already and you can simply run houdini from there if you want. If you are on linux it is the same. If you are on Windows then I imagine power shell or something? Sorry I ditched windows years ago and am not sure what would be the best way there.

[gzimmermann123]

so basically instead of pressing that shelf tool inside Houdini that the MLOPs developer provide, you stay outside of Houdini in a shell and re-produce what that shell tool does. If you look closely at the code of that shelf tool, it basically boils down to go over the requirements.txt file that ships with the MLOPs repo and (pip) installs each package. By doing that in the shell you see at what packages it breaks. For me it was the 2 I mentioned in my post. In the shell you can simply do the steps I mentioned to get around the problems. Once pip install is done with all the packages you can start houdidni and the MLOPs should find all required python packages.

[gzimmermann123]

so more concretely: use the houdini shell (outside of Houdini) or a system shell, cd to where you have the MLOPs repo downloaded and run:
pip install -r requirements.txt
Then see where it breaks! Maybe it's the same packages than for me.

[Ambrosiussen]

I have fixed the installing issues on Windows + Linux for H20. Please try now :)
If you are using older versions of Houdini please see: https://github.com/Bismuth-Consultancy-BV/MLOPs/releases

[MtDersvan]

Not working on Mac out of the box just yet.

Several lines need to change:

• rembg[gpu]==2.0.54 –> rembg==2.0.54 (gpu version is not supported)
• xformers==0.0.21 not supported
• chumpy @ git+https://github.com/mattloper/chumpy@51d5afd92a8ded3637553be8cef41f328a1c863a not supported
• /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk clang is hardcoded somewhere (probably in diffusers), so I had to manually link it to a newly installed /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
• --use-pep517 parameter is needed for proper installation of a few packages (e.g. basicsr==1.4.2), so you have to change the line in mlops_utils.py to cmd = ["hython", "-m", "pip", "install", "--use-pep517", "--target", os.path.join(PIP_FOLDER, "temp")]

This will result in Houdini saying dependency are installed correctly. That said, "Download Model" results in Houdini crash with a Load failed for /User/username/from multiprocessing.resource_tracker import main /User/username/main(31) Error window.

EDIT: From what I see, calling snapshot_download() from within Houdini crashes the application (at least opens up a new one), but I could run that code from a python shell no problem. I also see that Houdini triggers a separate process:

/Applications/Houdini/Houdini20.0.643/Houdini FX 20.0.643.app/Contents/MacOS/houdini -c from multiprocessing.resource_tracker import main;main(31)

That actually keeps running until the download is complete....