Is provider: BEDROCK meant to return Anthropic's Messages shape instead of the real AWS Converse API shape?

[silvi-t]

Context

I'm testing Kuadrant's TokenRateLimitPolicy (TRLP), which ships a built-in default dataExtraction JSON Pointer for AWS Bedrock Converse (/usage/totalTokens) - see the reference docs. I wanted to use MockServer's LLM response mocking to fabricate a Bedrock Converse response and verify that pointer resolves correctly, but the mocked response doesn't match AWS's real shape.

Issue

Mocking provider: BEDROCK at /model/{model}/converse returns a response in Anthropic's Messages API shape (snake_case usage.input_tokens/output_tokens, no total_tokens, content[].text, msg_... id) rather than
the real AWS Bedrock Converse API envelope (usage.inputTokens/outputTokens/totalTokens, output.message.content[]) that the LLM Response Mocking docs describe for this provider.

Repro expectation:

{
  "httpRequest": { "method": "POST", "path": "/model/amazon.titan-text-express-v1/converse" },
  "httpLlmResponse": {
    "provider": "BEDROCK",
    "model": "amazon.titan-text-express-v1",
    "completion": { "text": "mocked response", "usage": { "inputTokens": 5, "outputTokens": 5 } }
  }
}

Response received:

{
  "id": "msg_305fe52e4ae04909bc51bab8",
  "type": "message",
  "role": "assistant",
  "content": [ { "type": "text", "text": "mocked response" } ],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 5, "output_tokens": 5 }
}

This also lines up with the 7.5.0 release notes for LlmCodecGoldenFileTest.shouldEncodeCanonicalTokenUsageCounts, which list only 4 usage-schema patterns across 7 providers, with Anthropic and Bedrock sharing input_tokens/output_tokens - suggesting the Bedrock codec intentionally delegates to the Anthropic codec rather than emitting AWS's own camelCase Converse schema.

Question

Is this intentional? If so, could the docs be updated to say BEDROCK currently returns an Anthropic-compatible shape rather than the real Converse envelope? If not, could the Bedrock codec be corrected to emit the real AWS shape (usage.inputTokens/outputTokens/totalTokens, output.message.content[])?

[tk1475]

From the code, the Anthropic shape is what the codec is designed to return right now. The docs table is what's out of step. BedrockCodec says in its Javadoc that it targets "AWS Bedrock (Anthropic-on-Bedrock) invokeModel API (version bedrock-2023-05-31)", i.e. the plain Anthropic body that InvokeModel returns for Claude models. encode() just delegates to AnthropicCodec:

private final AnthropicCodec delegate = new AnthropicCodec();
...
public HttpResponse encode(Completion completion, String model) {
    return delegate.encode(completion, model);
}

So you get usage.input_tokens/output_tokens whatever the path or model is. Meanwhile, the LLM response mocking page lists BEDROCK as /model/{model}/converse, "AWS Bedrock Converse API". There's no Converse encoder in the codec today, so a docs fix (or a Converse codec) is worth an issue.

For testing the Kuadrant /usage/totalTokens pointer, you don't need httpLlmResponse at all. A plain httpResponse with a Converse-shaped body works:

{
  "httpRequest": { "method": "POST", "path": "/model/amazon.titan-text-express-v1/converse" },
  "httpResponse": {
    "statusCode": 200,
    "headers": { "content-type": ["application/json"] },
    "body": {
      "type": "JSON",
      "json": {
        "output": { "message": { "role": "assistant", "content": [ { "text": "mocked response" } ] } },
        "stopReason": "end_turn",
        "usage": { "inputTokens": 5, "outputTokens": 5, "totalTokens": 10 },
        "metrics": { "latencyMs": 100 }
      }
    }
  }
}

That gives you full control over totalTokens, which you'll want anyway to check the rate limit counting.

[jamesdbloom]

Thanks for the clear repro, and thanks @tk1475 for the accurate diagnosis.

This isn't intended. The docs promise the Bedrock Converse API for provider: BEDROCK, but the codec actually implements Bedrock InvokeModel for Anthropic Claude models. That API returns the plain Anthropic Messages body, so BedrockCodec hands encoding straight to the Anthropic codec. As a result you get usage.input_tokens / output_tokens and a top-level content[] whatever the path or model. You don't get the Converse envelope:

{
  "output": { "message": { "role": "assistant", "content": [ { "text": "..." } ] } },
  "stopReason": "end_turn",
  "usage": { "inputTokens": 5, "outputTokens": 5, "totalTokens": 10 },
  "metrics": { "latencyMs": 100 }
}

I'll treat this as a bug. The fix is to give BEDROCK a real Converse encoder (and ConverseStream events for streaming) that emits the camelCase inputTokens / outputTokens / totalTokens usage, since that is what the docs describe and what the AWS SDKs and tools like Kuadrant's default /usage/totalTokens pointer expect. Until that lands, I'll also correct the docs so they say what BEDROCK returns today.

For testing your TokenRateLimitPolicy now, the workaround above is the right one. A plain httpResponse with a Converse-shaped JSON body lets you set totalTokens exactly, which also makes it easy to check the rate-limit counting against known values.

We expect to cut a new release within the next 5 days, once we finish tuning a few final performance points, and the Converse fix will be in it.

[jamesdbloom]

The fix for this is now on master in afeada5f1.

With provider: BEDROCK, a mock on /model/{model}/converse now returns the real Converse envelope: output.message.content[], stopReason, usage.inputTokens / outputTokens / totalTokens and metrics. Your exact expectation from above returns usage.totalTokens: 10, so Kuadrant's default /usage/totalTokens pointer resolves. /model/{model}/converse-stream returns ConverseStream events (messageStart, contentBlockDelta, messageStop, metadata) in AWS event-stream framing. /invoke and /invoke-with-response-stream keep returning the InvokeModel body as before. Converse requests are now also understood by the conversation matchers.

It will be in the next release.

[silvi-t]

Thanks for the quick fix! Looking forward to the next release that includes this. In the meantime, I've worked around it in our testsuite by hand-crafting the Converse-shaped response body via a plain httpResponse expectation. Will switch back to the built-in provider mocking once the fix ships in a release we pick up.