Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ai-proxy): AWS Bedrock driver #13054

Closed
wants to merge 57 commits into from
Closed

feat(ai-proxy): AWS Bedrock driver #13054

wants to merge 57 commits into from

Conversation

tysoekong
Copy link
Contributor

@tysoekong tysoekong commented May 22, 2024 •
edited
Loading

Summary

This is a draft PR that adds AWS Bedrock "chat inference" driver support.

Now Kong users can call a massive array of Anthropic, Cohere, Mistral, Amazon, Meta, AI21, and loads more models; all using the same flat format.

This is missing streaming support - I need to write a C/Lua decoder for the AWS application/vnd.amazon.eventstream format, and need e.g. Thijs help.

Checklist

  • The Pull Request has tests
  • A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
  • There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Internal (KAI-38).

@tysoekong tysoekong changed the title Feat/ai proxy aws bedrock driver DRAFT: feat(ai-proxy): AWS Bedrock driver May 22, 2024
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock_driver branch from d7aa420 to 69cd75d Compare June 6, 2024 09:03
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock_driver branch from b84d205 to 567a9cb Compare June 28, 2024 16:10
@subnetmarco
Copy link
Member

@tysoekong Let's get this merged and remove the draft status.

@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock_driver branch from d78873c to f5c477c Compare July 8, 2024 18:52
@tysoekong tysoekong marked this pull request as ready for review July 8, 2024 18:52
@tysoekong tysoekong changed the title DRAFT: feat(ai-proxy): AWS Bedrock driver feat(ai-proxy): AWS Bedrock driver Jul 8, 2024
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock_driver branch from f5c477c to 006699b Compare July 9, 2024 21:14
@tysoekong
stash
e21348e
@tysoekong
working static format for vertex; patched broken streaming termination
b533e93
@tysoekong
finished gemini (text-only) support
d61109d
@tysoekong
entity catch on gemini
79d163b
@tysoekong
repair gemini split logic
081b117
@tysoekong
fix(ai-proxy): gemini streaming transformer bug
5cbf424
@tysoekong
working public and vertex gemini chat text
05346e4
@tysoekong
working static format for vertex; patched broken streaming termination
ed8e5d2
@tysoekong
finished gemini (text-only) support
1013714
@tysoekong
finished bedrock integration; todo minus streaming
ec964c1
@tysoekong
stash
72cd604
@tysoekong
feat(ai-proxy): added aws streaming format parser
a29d258
@tysoekong
stash
3957076
@tysoekong
feat(bedrock): complete
06d6c8f
@tysoekong
fix(*): bedrock has streaming token counts
d35dbbf
@tysoekong
fix(bedrock): streaming token counts
466d307
@tysoekong
fix(bedrock): attempt 1 to fix access key auth
413eed7
@tysoekong
stash
33f9516
@tysoekong
fix(*): bedrock signing with no session token
5405a79
@tysoekong
remove old debugging
eefc439
@tysoekong
stash
c5afec2
@tysoekong
fix(ai-proxy): bedrock streaming format detection failure
b9c6612
@tysoekong
fix(ai-proxy-bedrock): add toolConfig and additional fields to Bedroc…
3d0247e 
…k; fix bedrock converse system prompts
@tysoekong
fix(ai-proxy): move stream parsers to per-provider layout
ceb3dfd
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock_driver branch from 523af5b to ceb3dfd Compare July 9, 2024 21:27
@tysoekong tysoekong closed this Jul 9, 2024
@tysoekong
Copy link
Contributor Author

This branch is a complete mess of rebases, I'm going to start this again.

@tysoekong tysoekong mentioned this pull request Jul 9, 2024
Merged
3 tasks
fffonion pushed a commit that referenced this pull request Jul 30, 2024
@tysoekong
feat(ai-proxy): add AWS Bedrock Converse-API Driver (#13354)
264341d 
Supersedes #13054 which was completely broken.

Adds AWS Bedrock "Converse API" support to Kong AI Gateway.

AG-14
oowl pushed a commit that referenced this pull request Aug 15, 2024
@fffonion @tysoekong
feat(ai-proxy): add AWS Bedrock Converse-API Driver (#13354) (#9824)
61fc9ea 
Supersedes #13054 which was completely broken.

Adds AWS Bedrock "Converse API" support to Kong AI Gateway.

AG-14

Co-authored-by: Jack Tysoe <91137069+tysoekong@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tysoekong tysoekong

Labels
cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee plugins/ai-proxy plugins/ai-request-transformer plugins/ai-response-transformer schema-change-noteworthy size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tysoekong @subnetmarco