fix(bedrock): route Kimi/Moonshot models via bedrock-mantle OpenAI-compatible endpoint#13803
Closed
gopinaath wants to merge 4 commits into
Closed
fix(bedrock): route Kimi/Moonshot models via bedrock-mantle OpenAI-compatible endpoint#13803gopinaath wants to merge 4 commits into
gopinaath wants to merge 4 commits into
Conversation
…mpatible endpoint Kimi K2.5 on Bedrock via the Converse API has known tool call parsing bugs (vercel/ai#11409) where tool call markers leak into text output, causing premature end_turn and lost tool calls in multi-step agentic workflows. The same model works correctly through OpenAI-compatible endpoints. Route Moonshot AI models through Bedrock's OpenAI-compatible endpoint (bedrock-mantle.<region>.api.aws) instead of the Converse API. This bypasses the lossy Converse API translation layer entirely while keeping authentication and billing on Bedrock infrastructure. Validated with multi-turn tool calling tests: - Converse API: 2-3 tool calls then premature end_turn - bedrock-mantle: 15+ tool calls, no premature stopping Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
When no bearer token is available, use aws4fetch to sign requests to the bedrock-mantle endpoint with IAM credentials from the existing credential provider chain (env vars, CLI profiles, IAM roles, SSO). Bearer token still takes precedence when AWS_BEARER_TOKEN_BEDROCK is set. The SigV4 path reuses the same pattern as @ai-sdk/amazon-bedrock and the same credentialProvider already configured by OpenCode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bedrock-mantle endpoint requires credentials scoped to the 'bedrock-mantle' service, not 'bedrock'. Verified with live AWS endpoint which returns explicit error: "Credential should be scoped to correct service: 'bedrock-mantle'." Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without this flag, the OpenAI-compatible SDK omits
stream_options: {include_usage: true} from streaming requests,
causing bedrock-mantle to never return token usage data in the
stream. The auto-fix at getSDK doesn't help because Kimi's
api.npm is @ai-sdk/amazon-bedrock, not @ai-sdk/openai-compatible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
|
If we wanted to go this route we'd do it in models.dev instead, thanks for trying to help tho |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13807
Summary
bedrock-mantle.<region>.api.aws) instead of the Converse APIend_turnin multi-step agentic workflowsProblem
Kimi K2.5 on Bedrock via the Converse API exhibits premature stopping behavior — the model makes 2-3 tool calls then issues
end_turn, requiring multipleopencode runinvocations for tasks that should complete in one session. The same model via OpenAI-compatible endpoints chains 14+ tool calls autonomously.Root cause: Bedrock's Converse API has a known tool call parsing bug for Kimi/Moonshot models (vercel/ai#11409) where internal model tokens leak into text output instead of being parsed as tool calls. Additionally, Moonshot models are not listed in AWS's official Converse API tool calling support table.
Solution
When a Bedrock model ID contains "kimi" or "moonshot", route through
bedrock-mantle.<region>.api.aws/v1using@ai-sdk/openai-compatibleinstead of the Converse API. This:Validation
Multi-turn tool calling tests with 5 tools and a 4-step task:
Tested with both curl (15 rounds, never stopped) and AI SDK generateText (6 steps, 8 tool calls, task fully completed).
Test plan
amazon-bedrock/moonshotai.kimi-k2.5with multi-step tool calling task