[Bug]: Interrogate CLIP

[cerarslan]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

I looked at the problem here but found that the command lines are different. #5968

Steps to reproduce the problem

1. Upload image to img2img
2. Press ...interrogate CLIP

What should have happened?

Tensor dimension mismatch

Commit where the problem happens

Collab version.

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Brave

Command Line Arguments

load checkpoint from /content/gdrive/MyDrive/sd/stable-diffusion-webui/models/BLIP/model_base_caption_capfilt_large.pth
Error interrogating
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 148, in interrogate
    caption = self.generate_caption(pil_image)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 126, in generate_caption
    gpu_image = transforms.Compose([
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/transforms.py", line 95, in __call__
    img = t(img)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/transforms.py", line 270, in forward
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/functional.py", line 360, in normalize
    return F_t.normalize(tensor, mean=mean, std=std, inplace=inplace)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/functional_tensor.py", line 940, in normalize
    return tensor.sub_(mean).div_(std)
RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

Additional information, context and logs

No response

[DaveScream]

The same problem after git pull update :( also requirements wond install automatically. So i installed new lib by hands.

[ZioFester90]

i have the same problem after git pull. how to fix it? davescream how you installed new lib manually? can you tell me the steps please?

[FenrisValren]

Same after git pull. Interrogate just returning error

[cerarslan]

The same problem after git pull update :( also requirements wond install automatically. So i installed new lib by hands.

Can you please share steps?

[cibernicola]

Same here

[alcoartist]

same case

[allenbenz]

Looks like this was introduced with 9441c28

[SuperFurias]

i have the same problem
RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

[cerarslan]

Looks like this was introduced with 9441c28

Yes, you found that, it must be that but idk how csn we solve the problem.

[metixxx]

For a temporary solution, I reverted the changes that caused this problem to their previous state and it worked properly so that this problem can be solved correctly later.
You can replace these 4 files inside \modules folder (stable-diffusion-webui\modules) .

I put them in a zip file

fix-interrogate-error.zip

[allenbenz]

You can fix this by changing
prompt = shared.interrogator.interrogate(image) in modules/ui.py to
prompt = shared.interrogator.interrogate(image.convert("RGB"))

[ZioFester90]

thanks. it worked :D

[cerarslan]

still not working guys

Error interrogating
Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 148, in interrogate
caption = self.generate_caption(pil_image)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 133, in generate_caption
caption = self.blip_model.generate(gpu_image, sample=False, num_beams=shared.opts.interrogate_clip_num_beams, min_length=shared.opts.interrogate_clip_min_length, max_length=shared.opts.interrogate_clip_max_length)
File "/content/gdrive/MyDrive/sd/stablediffusion/src/blip/models/blip.py", line 156, in generate
outputs = self.text_decoder.generate(input_ids=input_ids,
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 1268, in generate
self._validate_model_kwargs(model_kwargs.copy())
File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 964, in _validate_model_kwargs
raise ValueError(
ValueError: The following model_kwargs are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)

Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 148, in interrogate
caption = self.generate_caption(pil_image)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 133, in generate_caption
caption = self.blip_model.generate(gpu_image, sample=False, num_beams=shared.opts.interrogate_clip_num_beams, min_length=shared.opts.interrogate_clip_min_length, max_length=shared.opts.interrogate_clip_max_length)
File "/content/gdrive/MyDrive/sd/stablediffusion/src/blip/models/blip.py", line 156, in generate
outputs = self.text_decoder.generate(input_ids=input_ids,
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 1268, in generate
self._validate_model_kwargs(model_kwargs.copy())
File "/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 964, in _validate_model_kwargs
raise ValueError(
ValueError: The following model_kwargs are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/gradio/routes.py", line 284, in run_predict
output = await app.blocks.process_api(
File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 982, in process_api
result = await self.call_function(fn_index, inputs, iterator)
File "/usr/local/lib/python3.8/dist-packages/gradio/blocks.py", line 824, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.8/dist-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/ui.py", line 273, in interrogate
prompt = shared.interrogator.interrogate(image.convert("RGB"))
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/interrogate.py", line 177, in interrogate
res += ""
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

[imranmu]

There is a solution link
It's the transformers library. Please check out the dev branch or modify the transformers line of extensions/sd_dreambooth_extension/requirements.txt to be

transformers==4.26.1