Route gemma models to mantle in bedrock#340
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7300af23a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| // `bedrock-runtime`: xAI/Grok (`xai.`) and Google Gemma 4 (`google.gemma`). | ||
| fn is_bedrock_mantle_only_model(model: &str) -> bool { | ||
| model.starts_with("xai.") | ||
| model.starts_with("xai.") || model.starts_with("google.gemma") |
There was a problem hiding this comment.
When an AWS-managed Bedrock endpoint is used with existing Gemma 3 IDs such as google.gemma-3-27b-it, this broad prefix now sends Chat Completions through mantle_openai_url, which hard-codes /openai/v1/chat/completions. AWS documents Gemma 3's mantle base as /v1, while the /openai/v1 mantle base is called out for Gemma 4/xAI, so this regresses Gemma 3 routing even though the change is meant for the new Gemma 4 family; narrow this to google.gemma-4- or model the per-family path difference.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
this is fine, because all of the gemma 3 models also support mantle: https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-google.html
also it looks like we are missing gemma 3 models from the model_list anyways, I'll seperately fix that
There was a problem hiding this comment.
actually good grief this is cursed. they support mantle but a different mantle endpoint, so its actually better for them to stay on runtime .... i'll just fix this so its not a problem when I fix the model lsit
2 participants
Bedrock introduced new gemma models that only route to mantle (https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-google-gemma-4-e2b.html), so we need to add them to our mantle-only list.
If this starts becoming more of an issue we can add it to the model config 🤕 but for now this is fine I think
Manually validated that this invokes e2e, we already have e2e mantle testing so I think that is sufficient.