-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question] Possibility for Multi-image input? #197
Comments
Hi @jsg921019 Thank you for your interest in our work. Due to the current way of training, we do not observe the model having very good capability referring to / comparing with multiple images. We are working on improving this aspect as well, stay tuned! You can also checkout some of the discussions here. |
This is also of great interest to our group as well. We work with pathology images and it can take more than one image to describe a region of interest due to image size. In this case we don't need to compare images, but allow several images to represent one thing. This would be similar in concept to MIL modeling (https://github.com/Project-MONAI/tutorials/tree/main/pathology/multiple_instance_learning). |
@codybum Thanks for explaining this interesting direction! I am curious about if there is any plan in your group in working in this direction? Would be happy to integrate that into LLaVA :) |
I am also looking forward to seeing this feature soon. For instance, GPT-4V can take several images and find relationships between objects on different images. It's pretty cool and has a variety of use cases. |
@haotian-liu In my understanding GPT4v slices higher resolution images into 512x512 images plus one context image and then tokenizes + collates those tokens. Have you tried something like this with the latest LLaVA model by any chance? Is it something worth trying? My use case is simlar to @codybum 's where I need to pass in higher resolution images. |
Question
I really enjoyed reading this paper and I have played with it few days and came up to this question:
LLava Architecture is capable of having more than one image input. I tried give two images as an input, but the inference result was not good. That is probably the model was not trained with multiple image input. Have you tried training LLaVA with dataset that has more than one input??
The text was updated successfully, but these errors were encountered: