-
-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enable max tolkens for anthropic/claude-3-sonnet-20240229 #1094
Comments
I have a suspicion that it's an issue on the LiteLLM-side, could you try isolating the issue by trying with LiteLLM only? Keep us updated! |
I'm not so sure that it's LiteLLM to blame here, I've tried now with an older version of it and the same behaviour is happening on Claude 2 as well, from other clients than WebUI. I believe this behavior started the day that Claude 3 was released. By chance @bjornjorgensen are you using an Anthropic developer account like mine? I am beginning to wonder if they simply limit |
Update: now I'm not so sure where the blame lies. Testing in another chat app works fine with Claude 3 endpoints. So it could indeed be a LiteLLM issue then, but it also wasn't working with older versions of it that previously did work? Really need someone with actual paid API keys to test this I think. |
Yes I just tried it with Lobechat as well and got a full response. So ball seems to be back in LiteLLM's court but I need to do further testing with components in isolation to be really certain of this. |
So after futher testing... I can only observe this happening when WebUI is involved. So it seems it may be something to do with our code, I just cannot at the moment nail down what it could possibly be. It seemingly only affects Claude API via LiteLLM in Open WebUI |
are there any configs that over rights tolkens that it can print on outputs? |
@bjornjorgensen nah, in our testing we've checked we're not sending anything like that would limit the max tokens, but nonetheless the API response says the stop reason is |
Ladies and gentleman, we got em. Claude's API now requires that the - litellm_params:
api_key: your_api_key
model: anthropic/claude-3-sonnet-20240229
max_tokens: 4096
model_info:
id: 810226a0-61e2-4d97-9de0-822bd4300fcd
model_name: claude-3-sonnet Note: the maximum value is 4096, you'll get an error from Anthropic's API if you request more. |
v0.1.111 (not merged to |
max_tokens: 4096 should be explicitly set from the settings! |
yes, I have to delete the old one and add it back.. but now it works :) |
@bjornjorgensen I haven't migrated this to the docs site yet, there's a thread:
|
Bug Report
Cant change max tolken when running anthropic/claude-3-sonnet-20240229
so the text is cut.
Description
Bug Summary:
[Provide a brief but clear summary of the bug]
Steps to Reproduce:
[Outline the steps to reproduce the bug. Be as detailed as possible.]
add anthropic/claude-3-sonnet-20240229
as for something bigger then 2 + 2 ?
Expected Behavior:
[Describe what you expected to happen.]
Actual Behavior:
[Describe what actually happened.]
Environment
breave
Reproduction Details
Confirmation:
Logs and Screenshots
Browser Console Logs:
[Include relevant browser console logs, if applicable]
Docker Container Logs:
[Include relevant Docker container logs, if applicable]
Screenshots (if applicable):
[Attach any relevant screenshots to help illustrate the issue]
Installation Method
[Describe the method you used to install the project, e.g., manual installation, Docker, package manager, etc.]
Additional Information
[Include any additional details that may help in understanding and reproducing the issue. This could include specific configurations, error messages, or anything else relevant to the bug.]
Note
If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!
The text was updated successfully, but these errors were encountered: