[demo] error encountered running Mini Demo code in https://github.com/Luodian/Otter/blob/main/docs/demo.md.

[laserwave]

File "Otter/Otter/xformers_model/clip.py", line 77, in forward
patch_embeds = self.patch_embedding(
File "python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: GET was unable to find an engine to execute this computation

Environment:
torch 2.0.0, python 3.9, cuda11.7,

[Luodian]

hi I seems the error with xformers, you could try to uninstall xformers and run it again?

pip uninstall xformers

And make sure you are using the latest main branch code.

[laserwave]

File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward
patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid]
File ".local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: GET was unable to find an engine to execute this computation

not work， similar error ，using the latest main branch code

[laserwave]

should i downgrade the torch version from 2.0.0 to 1.x

[Luodian]

should i downgrade the torch version from 2.0.0 to 1.x

I dont think so, we use torch 2.0 as well. Could you paste your accelerate package version? It's weird to see this error.

[Luodian]

@zhuwenzhen Please help to it, thanks!

[cliangyu]

I am not really sure if downgrading torch suite is a working solution chenfei-wu/TaskMatrix#283

But you can give it a shot for a quick possible fix : )

[Nntsyeo]

Great project! But Im stuck at this issue too. I've tried downgraded torch or uninstalled xformers. Couldn't work as well. Has this been resolved for @laserwave ?

[Luodian]

@Nntsyeo Do you have the same issue, can you post it so we can see what's the possible reason could be.

[Nntsyeo]

@Luodian Pardon me, here's my encounter.

Error:

(otter) user@env:~/Otter$ python test-model.py                             
                                                                                    
Using pad_token, but it is not set yet.                                            
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/
███████████████████████████████████████████████| 4/4 [00:35<00:00,  8.92s/it]       
Enter prompts (comma-separated): what are they doing?                              

Prompt: what are they doing?                                                       
Traceback (most recent call last):                                                 
  File "/home/user/Otter/test-model.py", line 141, in <module>                                                                                                        
    response = get_response(frames_list, prompt, model, image_processor)                                                                                              
  File "/home/user/Otter/test-model.py", line 98, in get_response                                                                                                     
    generated_text = model.generate(                                               
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context                                        
    return func(*args, **kwargs)                                                   
  File "/home/user/Otter/otter/modeling_otter.py", line 873, in generate                                                                                              
    self._encode_vision_x(vision_x=vision_x)                                       
  File "/home/user/Otter/otter/modeling_otter.py", line 831, in _encode_vision_x                                                                                      
    vision_x = self.vision_encoder(vision_x)[0][:, 1:, :]                                                                                                             
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl                                             
    return forward_call(*args, **kwargs)                                           
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 940, in forward                                  
    return self.vision_model(                                                      
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl                                             
    return forward_call(*args, **kwargs)                                           
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 865, in forward                                  
    hidden_states = self.embeddings(pixel_values)                                  
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl                                             
    return forward_call(*args, **kwargs)                                           
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward                                  
    patch_embeds = self.patch_embedding(pixel_values)  # shape = [*, width, grid, grid]                                                                               
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl                                             
    return forward_call(*args, **kwargs)                                           
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward                                                   
    return self._conv_forward(input, self.weight, self.bias)                                                                                                          
  File "/home/user/miniconda3/envs/otter/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward                                             
    return F.conv2d(input, weight, bias, self.stride,                              
RuntimeError: std::bad_alloc

Can I also check if this code (in main) remains correct for the demo?

    model = OtterForConditionalGeneration.from_pretrained(
        "luodian/otter-9b-dc-hf",
    )

My packages are:

(otter) user@env:~/Otter$ pip list | grep -e torch -e xformers
open-clip-torch          2.20.0
torch                    2.0.1
torchaudio               2.0.2
torchvision              0.15.2

[ZhangYuanhan-AI]

File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid] File ".local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: GET was unable to find an engine to execute this computation

not work， similar error ，using the latest main branch code

Compare cuda version returned by nvidia-smi and nvcc --version. They need to match. Or at least, the version get by nvcc --version should be <= the version get by nvidia-smi.

I think probably your GPU can not support cuda 11.7 and above. But torch 2.0.0 can only be installed with cuda 11.7 and above.

[laserwave]

File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid] File ".local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: GET was unable to find an engine to execute this computation
not work， similar error ，using the latest main branch code

Compare cuda version returned by nvidia-smi and nvcc --version. They need to match. Or at least, the version get by nvcc --version should be <= the version get by nvidia-smi.

I think probably your GPU can not support cuda 11.7 and above. But torch 2.0.0 can only be installed with cuda 11.7 and above.

NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-cabb626d-bd6f-51f7-c914-1ab6087e2de8)
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-5cbd302a-aabc-d2d2-9c06-3a0bf21910ea)
GPU 2: NVIDIA GeForce RTX 3090 (UUID: GPU-564415d2-a276-eb55-934c-1e15ec82a0a2)
GPU 3: NVIDIA GeForce RTX 3090 (UUID: GPU-14150f3a-f8ba-dd5a-7ac4-4cc361e80e7c)

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

[ZhangYuanhan-AI]

File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/transformers/models/clip/modeling_clip.py", line 195, in forward patch_embeds = self.patch_embedding(pixel_values) # shape = [*, width, grid, grid] File ".local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "miniconda3/envs/otter_py39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward output = old_forward(*args, **kwargs) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File ".local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: GET was unable to find an engine to execute this computation
not work， similar error ，using the latest main branch code

Compare cuda version returned by nvidia-smi and nvcc --version. They need to match. Or at least, the version get by nvcc --version should be <= the version get by nvidia-smi.
I think probably your GPU can not support cuda 11.7 and above. But torch 2.0.0 can only be installed with cuda 11.7 and above.

NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-cabb626d-bd6f-51f7-c914-1ab6087e2de8) GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-5cbd302a-aabc-d2d2-9c06-3a0bf21910ea) GPU 2: NVIDIA GeForce RTX 3090 (UUID: GPU-564415d2-a276-eb55-934c-1e15ec82a0a2) GPU 3: NVIDIA GeForce RTX 3090 (UUID: GPU-14150f3a-f8ba-dd5a-7ac4-4cc361e80e7c)

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0

Ok. Can you run smoothly for this code？

from PIL import Image
import requests

from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

[Luodian]

Hi it seems that another related issue has been addressed. #165