Add media support to vertex chat_models #227
Conversation
| # Handle media type like LangChain.js | ||
| # https://github.com/langchain-ai/langchainjs/blob/e536593e2585f1dd7b0afc187de4d07cb40689ba/libs/langchain-google-common/src/utils/gemini.ts#L93-L106 | ||
| if part["type"] == "media": | ||
| if "mime_type" not in part: |
There was a problem hiding this comment.
I went with snake_case for mime_type and file_uri contrary to what's happening in langchain.js as it seemed more pythonic. Do you think that's the right call?
There was a problem hiding this comment.
can we construct the proto Part directly? why do we need a VertexPart?
and it's probably a good idea to allow to pass gcs_uri as file_data and video_metadata too.
There was a problem hiding this comment.
can we construct the proto Part directly?
Done.
video_metadata
Done.
pass gcs_uri
file_uri expects a GCS uri, e.g. "gs://bucket/some_video.mp4". Do you mean we should also support http uri-s here?
There was a problem hiding this comment.
so there's no way to send the video into a base64 uri?, instead of an GCS uri?
| llm = ChatVertexAI(model_name="gemini-1.5-pro-preview-0514") | ||
| storage_client = storage.Client() | ||
| blob = storage.Blob.from_string(file_uri, client=storage_client) | ||
| media_base64 = base64.b64encode(blob.download_as_bytes()).decode() |
There was a problem hiding this comment.
could you include a test that passes a gcs_uri, please?
There was a problem hiding this comment.
See test_multimodal_media_file_uri.
Fixes langchain-ai/langchain#20738.