You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, I am not sure if this is an issue related to lmql decoder ot the underlying model itself.
I am running as a server an instance of Mistral (llama.cpp format) with the following params:
[Loading llama.cpp model from llama.cpp:/home/alexander/Games2/lmql/models/mistral-7b-v0.1.Q5_K_M.gguf with {'n_ctx': 4096, 'n_gpu_layers': 35, 'repeat_penalty': 1.2, 'temp': 0.8, 'device_map': 'auto'} ]
My client in the playground has the following lmql code taken from the examples in the docs:
argmax
# review to be analyzed
review = """We had a great stay. Hiking in the mountains was fabulous and the food is really good."""
# use prompt statements to pass information to the model
"Review: {review}"
"Q: What is the underlying sentiment of this review and why?"
# template variables like [ANALYSIS] are used to generate text
"A:[ANALYSIS]" where not "\n" in ANALYSIS
# use constrained variable to produce a classification
"Based on this, the overall sentiment of the message can be considered to be[CLS]"
from
lmql.model("llama.cpp:/home/alexander/Games2/lmql/models/mistral-7b-v0.1.Q5_K_M.gguf",
endpoint="localhost:9999",
tokenizer="mistralai/Mistral-7B-v0.1")
distribution
CLS in [" positive", " neutral", " negative"]
Then the model keeps repeating over and over the same thing.
Review: We had a great stay. Hiking in the mountains was fabulous and the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food is really good.Q: What is the underlying sentiment of this review and why?A: The underlying sentiment of this review is that the food
Is this related to the argmax function and "where not "\n"" part ?
Thank you for your help.
Best Regards
Aleks
The text was updated successfully, but these errors were encountered:
Hello, I am not sure if this is an issue related to lmql decoder ot the underlying model itself.
I am running as a server an instance of Mistral (llama.cpp format) with the following params:
[Loading llama.cpp model from llama.cpp:/home/alexander/Games2/lmql/models/mistral-7b-v0.1.Q5_K_M.gguf with {'n_ctx': 4096, 'n_gpu_layers': 35, 'repeat_penalty': 1.2, 'temp': 0.8, 'device_map': 'auto'} ]
My client in the playground has the following lmql code taken from the examples in the docs:
Then the model keeps repeating over and over the same thing.
Is this related to the argmax function and "where not "\n"" part ?
Thank you for your help.
Best Regards
Aleks
The text was updated successfully, but these errors were encountered: