Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vertexai gemini has a different URL, we should support that in the vertexai procedure #3939

Closed
jexp opened this issue Feb 5, 2024 · 1 comment

Comments

@jexp
Copy link
Member

jexp commented Feb 5, 2024

see:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    -d @gemini.json \ 
    "https://us-central1-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/us-central1/publishers/google/models/gemini-pro:streamGenerateContent"

currently we use :predict as suffix which doesn't seem to work anymore with gemini, also the request body changed a bit

we can probably have a new method called apoc.ml.vertexai.stream

that also works with the new body.

gemini-pro-vision supports also image payloads to e.g. describe or summarize images, not sure if we need that yet

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#request_body

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-pro-vision:streamGenerateContent

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#gemini-pro-vision

We probably need that for gemini ultra anyhow.

@jexp
Copy link
Member Author

jexp commented Feb 5, 2024

Perhaps in addition we should add a generic URL method like we did in bedrock that allows to use more custom new model URLs

and perhaps an Image Vision procedure as well as we did for Bedrock.

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/image-generation

it seems to support a GCS storage URL as optional parameter, so that the generated images are stored in GCS and an URL returned, we should support that.

And also see if something similar is available for the bedrock image-gen API with S3 and add it there too.

vga91 added a commit that referenced this issue Feb 14, 2024
vga91 added a commit that referenced this issue Feb 14, 2024
vga91 added a commit that referenced this issue Feb 14, 2024
vga91 added a commit that referenced this issue Feb 16, 2024
vga91 added a commit that referenced this issue Feb 19, 2024
vga91 added a commit that referenced this issue Feb 23, 2024
vga91 added a commit that referenced this issue Feb 23, 2024
…hat in the vertexai procedure (#3947)

* Fixes #3939: vertexai gemini has a different URL, we should support that in the vertexai procedure

* updated extended.txt

* changed endpoint from String.format to string replace with parameter - updated image and assertion

The previoous image was sometimes recognized as a `bagel`,
and changed assertion from `tarallo` to `tarall` since the output can be both `tarallo` and `taralli`

* Fix compile error after #3940 merge

* changed %2s to {project} in docs
@vga91 vga91 closed this as completed Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

3 participants