[Usage] Is there a way to describe one image and the next?

[min731]

Describe the issue

Thanks for your great work!!

I want to load the model and input multiple images to save the description for each image.
Is there a way to load the model and then complete the description for one image and then have it describe for the next one?

----- file 1 inference ------
ASSISTANT: Yes
ASSISTANT: Sunset, beach, ocean, car, man, woman, boy, girl, handbag, backpack, cell phone.
ASSISTANT: A family photo taken in front of a sea of waves.
inference_outputs : ['Yes', 'Sunset, beach, ocean, car, man, woman, boy, girl, handbag, backpack, cell phone.', 'A family photo taken in front of a sea of waves.']
------ file 2 inference ------
[2023-10-18 17:06:24,340] ERROR in app: Exception on /album_registration_pc1/images_analysis_pc1 [POST]
Traceback (most recent call last):
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/flask/app.py", line 1455, in wsgi_app
response = self.full_dispatch_request()
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/flask/app.py", line 869, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
rv = self.dispatch_request()
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/home/meta-3/jm/META_FINAL_PJT/jm_PC1/Family_Album_PC1/views/album_registration_views_pc1.py", line 27, in images_analysis_task_pc1
inference_outputs = image_processor_pc1.llava_inference_image(
File "/home/meta-3/jm/META_FINAL_PJT/jm_PC1/Family_Album_PC1/models/image_processor_pc1.py", line 28, in llava_inference_image
inference_outputs = inference_image(tokenizer,
File "/home/meta-3/jm/META_FINAL_PJT/jm_PC1/Family_Album_PC1/models/LLaVA/llava/serve/cli.py", line 149, in inference_image
output_ids = model.generate(
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 1588, in generate
return self.sample(
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 2642, in sample
outputs = self(
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/meta-3/anaconda3/envs/llava/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/meta-3/jm/META_FINAL_PJT/jm_PC1/Family_Album_PC1/models/LLaVA/llava/model/language_model/llava_llama.py", line 75, in forward
input_ids, attention_mask, past_key_values, inputs_embeds, labels = self.prepare_inputs_labels_for_multimodal(input_ids, attention_mask, past_key_values, labels, images)
File "/home/meta-3/jm/META_FINAL_PJT/jm_PC1/Family_Album_PC1/models/LLaVA/llava/model/llava_arch.py", line 129, in prepare_inputs_labels_for_multimodal
cur_image_features = image_features[cur_image_idx]
IndexError: index 1 is out of bounds for dimension 0 with size 1

I had the same error as above.

[wanxingDaze]

Hello, I also encountered the same problem when running multiple images, I wonder if you have solved it

File "/home/llmnav/kunyuwangv2/LLaVA/llava/serve/main.py", line 177, in
main(args)
File "/home/llmnav/kunyuwangv2/LLaVA/llava/serve/main.py", line 146, in main
output_ids = model.generate(
File "/home/llmnav/anaconda3/envs/llava/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/llmnav/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 1588, in generate
return self.sample(
File "/home/llmnav/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/generation/utils.py", line 2642, in sample
outputs = self(
File "/home/llmnav/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/llmnav/anaconda3/envs/llava/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/llmnav/kunyuwangv2/LLaVA/llava/model/language_model/llava_llama.py", line 75, in forward
input_ids, attention_mask, past_key_values, inputs_embeds, labels = self.prepare_inputs_labels_for_multimodal(input_ids, attention_mask, past_key_values, labels, images)
File "/home/llmnav/kunyuwangv2/LLaVA/llava/model/llava_arch.py", line 141, in prepare_inputs_labels_for_multimodal
cur_image_features = image_features[cur_image_idx]
IndexError: index 1 is out of bounds for dimension 0 with size 1

[min731]

Yeah, that's the problem I'm running into.
Now, I'm trying to use 'LLaVA/llava/serve/cli.py' file to have the model describe multiple images each, but it's hard to implement.

[adrielkuek]

Not sure if this is still an issue, but the fix to this is to simply reset the conversation at the start of the loop so that it doesn't continuously collect and append chat histories. Reset using this line: conv = conv_templates[args.conv_mode].copy() will do the job.

[min731]

Oh, my God. It was just a matter of resetting the conv. Thank you so much. God bless you...