examples/server -n -1 never stops generating

[felrock]

I'm trying to get the same output from examples/server as from examples/main, but without any luck. Having -n -1 as param in for examples/server just makes it generate the whole ctx. While in the examples/main the generator stops when it has generated a good answer it seems. Does it have to do with stop_words? Any ideas are welcome, thanks!

[coezbek]

I don't think the server supports the command line option -n. Rather you pass n_predict as an option when calling the completion endpoint. See the docs here:

https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md

There are two things which come to mind which you might be missing:

• Did you use a model which has the EOS token defined correctly?
• Did you set the stopping words using the stop option? I commonly use ["<|im_end|>", "</s>"]