ollama run qwen:0.5B, Reply exception, stuck in a loop.

[samzong]

>>> /show info
Model details:
Family              qwen2
Parameter Size      620M
Quantization Level  Q4_0

~ uname -m -s -r
Darwin 23.3.0 arm64

20240208104056.mp4

/label bug

[bm777]

I had the same behavior with phi2 model. I noticed that the model gives the right or the expected answer before going to a new line (\n). So I had to add "\n" in the stop list.

const stream = await generate({
        model: "phi",
        prompt: text,
        stream: true,
        options: {
            num_predict: 70,
            temperature: 0.65,
            penalize_newline: true,
            top_p: 0.9,
            // presence_penalty: 0.6,
            stop: ["\n", "User:", "Assistant:", "User:"] //["\n"]
        }
    })

It still cuts at a wrong place sometimes, but I can manage to just remove the words after the last punctuation: . or ,
This method will not work if the user ask for a list as a result (give me 3 recipes of cappuccino) -> then after generating the first, the model will try to add a new line for the second element of the list, and it becomes more complicated to control the level.
(any workaround for this use case?)

[jmorganca]

The infinite generation should be fixed now. As for the poor responses from smaller models – this may be from the prompt template, prompt or other reasons – all of which we are trying to improve.

[bm777]

@jmorganca So we should update the Ollama binary verison then right ?