(Dev Branch) ModuleNotFoundError: No module named 'models.clipseg'

[discordinated]

Describe your environment

• GPU: [mps]
• VRAM: [32GB]
• CPU arch: [arm]
• OS: [macOS]
• Python: [miniconda]
• Branch: [development]
• Commit: [Merge: 3a1003f e6fc8af]

Describe the bug
Pulled latest update to development, get no models.clipseg error when trying to launch invoke.py

To Reproduce
Steps to reproduce the behavior:

1. On MPS update to latest development branch
2. Run scripts/invoke.py
3. Wati for terminal to do it's dance
4. See error

(invokeai) MACHINENAME InvokeAI % python3 scripts/invoke.py

• Initializing, be patient...
  Traceback (most recent call last):
  File "/Users/USERNAME/InvokeAI/scripts/invoke.py", line 708, in
  main()
  File "/Users/USERNAME/InvokeAI/scripts/invoke.py", line 36, in main
  from ldm.generate import Generate
  File "/Users/USERNAME/InvokeAI/ldm/generate.py", line 37, in
  from ldm.invoke.txt2mask import Txt2Mask, SegmentedGrayscale
  File "/Users/USERNAME/InvokeAI/ldm/invoke/txt2mask.py", line 32, in
  from models.clipseg import CLIPDensePredT
  ModuleNotFoundError: No module named 'models.clipseg'

Expected behavior
invoke.py to work with no clipseg error

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[Fortyseven]

Adding to this, I saw git+https://github.com/invoke-ai/clipseg.git#egg=clipseg added to the dependencies and added it manually but no dice. (Then went through the usual process with the rest of the deps, but same result.)

(Ubuntu 22.04, Python 3.10.6, etc.)

[lstein]

Try this:

rm -rf src
conda env update -f environment-mac.yml
python scripts/preload_models.py

Removing src prevents frequent conda hangs

I just downgraded to main, rebuilt the environment to remove the clipseg dependency, then upgraded to development and ran conda env update, and all seems to be well.

[discordinated]

Same problem I'm afraid : Full traceback from console :

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/USERNAME/InvokeAI/scripts/invoke.py:708 in <module>                                             │
│                                                                                                  │
│   705 │   completer.set_line(cmd)                                                                │
│   706                                                                                            │
│   707 if __name__ == '__main__':                                                                 │
│ ❱ 708 │   main()                                                                                 │
│   709                                                                                            │
│                                                                                                  │
│ /Users/USERNAME/InvokeAI/scripts/invoke.py:36 in main                                                  │
│                                                                                                  │
│    33 │   │   sys.exit(-1)                                                                       │
│    34 │                                                                                          │
│    35 │   print('* Initializing, be patient...')                                                 │
│ ❱  36 │   from ldm.generate import Generate                                                      │
│    37 │                                                                                          │
│    38 │   # these two lines prevent a horrible warning message from appearing                    │
│    39 │   # when the frozen CLIP tokenizer is imported                                           │
│                                                                                                  │
│ /Users/USERNAME/InvokeAI/ldm/generate.py:37 in <module>                                                │
│                                                                                                  │
│     34 from ldm.invoke.devices import choose_torch_device, choose_precision                      │
│     35 from ldm.invoke.conditioning import get_uc_and_c                                          │
│     36 from ldm.invoke.model_cache import ModelCache                                             │
│ ❱   37 from ldm.invoke.txt2mask import Txt2Mask, SegmentedGrayscale                              │
│     38                                                                                           │
│     39 def fix_func(orig):                                                                       │
│     40 │   if hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():              │
│                                                                                                  │
│ /Users/USERNAME/InvokeAI/ldm/invoke/txt2mask.py:32 in <module>                                         │
│                                                                                                  │
│    29                                                                                            │
│    30 import torch                                                                               │
│    31 import numpy as  np                                                                        │
│ ❱  32 from models.clipseg import CLIPDensePredT                                                  │
│    33 from einops import rearrange, repeat                                                       │
│    34 from PIL import Image                                                                      │
│    35 from torchvision import transforms                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'models.clipseg'
(invokeai) MACHINENAME InvokeAI % 

[lstein]

Could you look in src and see if clipseg is installed?

[discordinated]

I see a folder for 'clip' in there but not 'clipseg'

[lstein]

Good, that's the first step in figuring out what's going wrong (it should be there). Is there a reference to clipseg in the environment file you're loading?

[discordinated]

From environment-mac.yml? If so, nope.

name: invokeai
channels:
  - pytorch
  - conda-forge
dependencies:
  - python==3.9.13
  - pip==22.2.2

  # pytorch left unpinned
  - pytorch==1.12.1
  - torchvision==0.13.1

  # I suggest to keep the other deps sorted for convenience.
  # To determine what the latest versions should be, run:
  #
  # ```shell
  # sed -E 's/ldm/ldm-updated/;20,99s/- ([^=]+)==.+/- \1/' environment-mac.yml > environment-mac-updated.yml
  # CONDA_SUBDIR=osx-arm64 conda env create -f environment-mac-updated.yml && conda list -n ldm-updated | awk ' {print "  - " $1 "==" $2;} '
  # ```
  - albumentations==1.2.1
  - coloredlogs==15.0.1
  - einops==0.4.1
  - grpcio==1.46.4
  - humanfriendly==10.0
  - imageio==2.21.2
  - imageio-ffmpeg==0.4.7
  - imgaug==0.4.0
  - kornia==0.6.7
  - mpmath==1.2.1
  - nomkl=1.0
  - numpy==1.23.2
  - omegaconf==2.1.1
  - openh264==2.3.0
  - onnx==1.12.0
  - onnxruntime==1.12.1
  - pudb==2022.1
  - pytorch-lightning==1.7.5
  - scipy==1.9.1
  - streamlit==1.12.2
  - sympy==1.10.1
  - tensorboard==2.10.0
  - torchmetrics==0.9.3
  - pip:
      - flask==2.1.3
      - flask_socketio==5.3.0
      - flask_cors==3.0.10
      - dependency_injector==4.40.0
      - eventlet==0.33.1
      - opencv-python==4.6.0
      - protobuf==3.19.5
      - realesrgan==0.2.5.0
      - send2trash==1.8.0
      - test-tube==0.7.5
      - transformers==4.21.2
      - torch-fidelity==0.3.0
      - -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
      - -e git+https://github.com/openai/CLIP.git@main#egg=clip
      - -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k_diffusion
      - -e git+https://github.com/TencentARC/GFPGAN.git#egg=gfpgan
      - -e .
variables:
  PYTORCH_ENABLE_MPS_FALLBACK: 1

[lstein]

Ack! That's my bad. Please cut-and-paste the clipseg line from environment.yml, and do a conda update. That should fix it.

Update: Weird. The clipseg line is in my version of the file, and I'm quite sure I committed and pushed it.

[discordinated]

Added - -e git+https://github.com/invoke-ai/clipseg.git#egg=clipseg

trying it now. Strangely my local install is missing that from environment.yml too so I've manually copied it from the github. Maybe I need to nuke it all and start over if this doesn't work :)

[discordinated]

That worked ! Thanks again. Seems I might have got stuck with something local screwing it up, as I had to stash changes to change branch. Anyway working again now.

[Fortyseven]

In my case, I do have a src/clipseg directory (with a weights dir and clipseg_weights.zip), and I've re-installed using requirements.txt. (And I do have the line in my requirements.txt.)

--prefer-binary

albumentations
einops
huggingface-hub
imageio-ffmpeg
imageio
kornia
# pip will resolve the version which matches torch
numpy
omegaconf
opencv-python
pillow
pip>=22
pudb
pytorch-lightning
scikit-image>=0.19
streamlit
pyreadline3
# "CompVis/taming-transformers" IS NOT INSTALLABLE
# This is a drop-in replacement
taming-transformers-rom1504
test-tube
torch-fidelity
torchmetrics
transformers==4.21.*
flask==2.1.3
flask_socketio==5.3.0
flask_cors==3.0.10
flaskwebgui==0.3.7
send2trash
dependency_injector==4.40.0
eventlet
realesrgan
git+https://github.com/openai/CLIP.git@main#egg=clip
git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k-diffusion
git+https://github.com/TencentARC/GFPGAN.git#egg=gfpgan
git+https://github.com/invoke-ai/clipseg.git#egg=clipseg

[Fortyseven]

I'll nuke and start fresh just in case. :P

[Fortyseven]

I noticed this while running preload_models.py:

  File "./scripts/preload_models.py", line 123
    model.load_state_dict(
                         ^
SyntaxError: '(' was never closed

Fixing that let it run successfully, but that doesn't fix the original issue on this end.

I nuked my virtual environment, and reinstalled dependencies from scratch. Still no dice.

I'll keep poking at it.

[Fortyseven]

Quick hack: if I change both references of from models.clipseg import CLIPDensePredT to from clipseg.clipseg import CLIPDensePredT, it works. 😛 👍