Getting code indexing error when using litellm with vertexai gemini-embedding-001 model

[hsuyuming]

App Version

3.21.5 (653104b)

API Provider

OpenAI Compatible

Model Used

gemini-embedding-001

🔁 Steps to Reproduce

1. OS: Google CloudWorkstation (linux), extension version: 3.31.5
2. I setup a litellm proxy with vertexai gemini-embedding-001 model within config.

general_settings:
  otel: false
litellm_settings:
  turn_off_message_logging: true
model_list:
  - model_name: gemini-embedding-001
    litellm_params:
      model:  vertex_ai/gemini-embedding-001
      vertex_project: <project>
      vertex_location: <location>
      drop_params: true

3. For embedding provider "OpenAI Compatible", Base URL: "http://0.0.0.0:4000/v1", API KEY: LITELLM APIKEY, Model: "gemini-embedding-001", Embedding Dimension: 3072
4. Error message:

litellm.exceptions.BadRequestError: litellm.BadRequestError: VertexAIException BadRequestError - {
  "error": {
   "code": 400,
   "message": "Unable to submit request because it included too many instances. Reduce the number of instances and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings: Unable to submit request because batchSize must be 1 but the entered value was 56. Update the batchSize value and try again.",
   "status": "INVALID_ARGUMENT"
  }
}
. Received Model Group=gemini-embedding-001

Related document:

• https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings

💥 Outcome Summary

Would expect we are able to use gemini-embedding-001 for embedding model

📄 Relevant Logs or Errors (Optional)

Error message:

litellm.exceptions.BadRequestError: litellm.BadRequestError: VertexAIException BadRequestError - {
  "error": {
   "code": 400,
   "message": "Unable to submit request because it included too many instances. Reduce the number of instances and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings: Unable to submit request because batchSize must be 1 but the entered value was 56. Update the batchSize value and try again.",
   "status": "INVALID_ARGUMENT"
  }
}
. Received Model Group=gemini-embedding-001

[dosubot]

Hi @hsuyuming! I'm Dosu and I’m helping the Roo-Code team.

The error happens because VertexAI's gemini-embedding-001 embedding model only supports a batch size of 1, but Roo Code (via LiteLLM) is sending larger batches. Based on current documentation, there isn't a documented way in Roo Code or LiteLLM to configure or limit the batch size for embedding requests, nor are there known workarounds or advanced settings for this scenario. The main configuration for model behavior is managed on the LiteLLM server, and Roo Code acts as a client to it docs.

If possible, try to limit the number of inputs per embedding request in your client code to one at a time. Otherwise, you may want to monitor release notes or documentation updates for future support of this use case, as this is a known gap in the current documentation.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[daniel-lxs]

Thanks for the detailed report.

It looks like the gemini-embedding-001 model requires a batchSize of 1, which could be causing the error you're seeing. Since Roo Code doesn't currently support customizing the batch size for embedding models, this model might not be compatible with the current setup.

You might want to check newer embedding models from Google.

Let me know if switching to newer models works for you.

[hsuyuming]

Hi @daniel-lxs Isn't gemini-embedding-001 the latest model? (https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings)

[daniel-lxs]

@hsuyuming
Try with this experimental model: https://ai.google.dev/gemini-api/docs/models#gemini-embedding