[Feature]: Cloudflare AI Gateway support for Bedrock

[Manouchehri]

The Feature

The docs aren't added yet, but Cloudflare has added AI Gateway support for Amazon Bedrock.

https://developers.cloudflare.com/ai-gateway/providers/

The example request looks like:

curl -X POST https://gateway.ai.cloudflare.com/v1/account_id_here/gateway_name_here/aws-bedrock/bedrock-runtime/us-east-1/model/amazon.titan-embed-text-v1/invoke \
  -v --aws-sigv4 aws:amz:us-east-1:bedrock \
  -H 'Accept: */*' \
  -H 'Content-Type: application/json' \
  -u key:secret \
  -d '{
    "prompt": "Where did the phrase Hello World come from"
  }'

Motivation, pitch

Free caching :D

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

[Manouchehri]

Lower priority since we aren't using Amazon Bedrock as much after discovering that our AWS Activate credits don't apply to Anthropic or Cohere models on Bedrock. =P

[Manouchehri]

Small note: I'm not even 100% sure how to use Cloudflare AI Gateway with Bedrock (ignoring LiteLLM), so don't worry about this ticket at all in the meantime. Not a high priority at all imo.

[Manouchehri]

This is doable using boto3 hooks. I helped Cloudflare figure this one out, they will eventually be documenting it at https://developers.cloudflare.com/ai-gateway/providers/bedrock/

In the meantime, gonna save my notes here just in case.

https://stackoverflow.com/a/66097613/2079814

event_system.register_first('after-sign.*.*', _add_header)

or:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#before-send

before-send looks like it'd work too.

Unlike my example, you should only only boto3 requests going to bedrock invoke and invoke stream, since hooking others is just gonna break things if eventually the same boto3 client is used to do other API calls in the future.

[Manouchehri]

JavaScript/TypeScript solution here: https://gist.github.com/Manouchehri/d1fa65b6d842a76d836cb6b3f9e1bbff

[Manouchehri]

Was added a long time ago.