Chat with ollama/mistral-7b behind litellm returns strange answer #2259
francesco086
started this conversation in
LLM Usage | 语言模型研究
Replies: 2 comments
-
Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible. |
Beta Was this translation helpful? Give feedback.
0 replies
-
@francesco086 I think it might be the issue of parameter. Try to add a high value of |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
💻 Operating System
Other
📦 Environment
Docker
🌐 Browser
Other
🐛 Bug Description
I serve the mistral-7b model using ollama, and set a litellm proxy in front of it.
I am, for example, able to run the command:
and get the expected response
I setup lobechat to use several OpenAI models via litellm (gpt 3.5, 4, and dalle3), and everything works fine. However, with ollama/mistral-7b I get the following behaviour (I pressed the "Stop" button after a while because it was too slow):
![Screenshot 2024-03-18 at 17 41 17](https://private-user-images.githubusercontent.com/8321888/313764171-bcac4a26-cb31-4403-b8a5-1a1f4c6da092.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTkwMjAzODgsIm5iZiI6MTcxOTAyMDA4OCwicGF0aCI6Ii84MzIxODg4LzMxMzc2NDE3MS1iY2FjNGEyNi1jYjMxLTQ0MDMtYjhhNS0xYTFmNGM2ZGEwOTIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjJUMDEzNDQ4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NjkxMDk4YWMyMDE0YjI2OTk4M2ZhMzhmZmZmMjgyYjQ0YzZjM2M0Mzk2Yjk2MzFhYThiYTE5ODY3NGY2MWQyOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.rYs7L8kWqC03LhtypdCxzpCi6RyFdwoafvu_2mvUELs)
🚦 Expected Behavior
No response
📷 Recurrence Steps
No response
📝 Additional Information
Services are running on Kubernetes, setup via ArgoCD.
Beta Was this translation helpful? Give feedback.
All reactions