Add support for multimodal embeddings from Google Vertex AI

[SDonkelaarGDD]

Feature request

Currently no support for multi-modal embeddings from VertexAI exists. However, I did stumble upon this experimental implementation of GoogleVertexAIMultimodalEmbeddings in LangChain for Javascript. Hence, I think this would also be a very nice feature to implement in the Python version of LangChain.

Motivation

Using multi-modal embeddings could positively affect applications that rely on information of different modalities. One example could be product search in a web catalogue. Since more cloud providers are making endpoints for multi-modal embeddings available, it makes sense to incorporate these into LangChain as well. The embeddings of these endpoints could be stored in vector stores and hence be used in downstream applications that are built using LangChain.

Your contribution

I can contribute to this feature.

[beatgeek]

It seems it would be an update to this - libs/langchain/langchain/embeddings/vertexai.py where currently it's only for textembedding-gecko and lacks the function for image embeddings.

[SDonkelaarGDD]

@beatgeek indeed, I agree that that could be a good place to put it. We could make an extra class method embed_multimodal() that calls the multi-modal endpoint, as it can be used to embed both the images and text.

[pmoraesm]

Indeed, it's a nice feature to have!

[dosubot]

Hi, @SDonkelaarGDD! I'm helping the LangChain team manage their backlog and am marking this issue as stale.

It looks like you opened this issue to request support for multi-modal embeddings from Google Vertex AI in the Python version of LangChain. There was some discussion in the comments about updating the vertexai.py file to include support for image embeddings, and you and others expressed interest in contributing to the implementation.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!