[Feature] Support vLLM with Anthropic API endpoints #238
f0o
started this conversation in
Feature Requests
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
vLLM supports both OpenAPI (/v1/chat) and Anthropic (/v1/messages) APIs. (vllm-project/vllm#21313)
Subjectively the Anthropic endpoint feels more responsive with SSE Streaming which could improve User Experience.
Adding an additional switch to the custom provider UI to select between OpenAPI and Anthropic style APIs would fully satisfy this Feature Request (at least front-facing).
Beta Was this translation helpful? Give feedback.
All reactions