Prompt flags when using interactive mode

[metaphier]

Hello, thanks for creating this cool project.

Was wondering if anyone else has been able to get system prompts to work in interactive mode?

When I use .\build\bin\Release\main -m quantized\model.bin --sys_file systemprompt.txt -i

the model doesn't seem to have awareness of the system prompt. Same if i use the flag like this without a separate file-s "System prompt goes here"

When removing the -i interactive flag, I am able to at least prompt the model with -p and --prompt_file, but this only returns a single response and I'd like to utilize the interactive mode if possible.

Any help would be appreciated!

[metaphier]

I should add that the two models I've tried are qwen2-0.5b and gemma2-2b quantized models downloaded using the model_downloader.py script. Perhaps it's a model specific issue?

[foldl]

-p is ignored in interactive mode, but -s is always used.

main.exe -m quantized\qwen2.5-1.5b.bin -s "you are a helpful assistant. always answer like Jack Sparrow, and use emojis" -i

    ________          __  __    __    __  ___ (通义千问)
   / ____/ /_  ____ _/ /_/ /   / /   /  |/  /_________  ____  
  / /   / __ \/ __ `/ __/ /   / /   / /|_/ // ___/ __ \/ __ \ 
 / /___/ / / / /_/ / /_/ /___/ /___/ /  / // /__/ /_/ / /_/ / 
 \____/_/ /_/\__,_/\__/_____/_____/_/  /_(_)___/ .___/ .___/  
You are served by QWen2,                      /_/   /_/       
with 1543714304 (1.5B) parameters.

You  > hi
A.I. > Ahoy there, matey! How be ye today? 🏺✨

You  > which one is larger, 9.8 or 9.11?
A.I. > Ahoy, matey! Let's see 'em! 9.8 is the bigger one, like a big, strong pirate ship! 🏆

[metaphier]

I tried it with qwen2.5-1.5b.bin and got similar results as you indicating it included the system prompt in the generation. However the -s flag for at least qwen2-0.5b and gemma2-2b still does not work based on my testing.

[foldl]

Gemma 1 & 2 do not support system prompt: https://huggingface.co/google/gemma-2-2b-it/blob/main/tokenizer_config.json#L2003

qwen2-0.5b is too small to act according to system prompt. You can try a larger one like 1.5b. By the way, qwen2.5 is definitely better than qwen2.

[foldl]

Closed due to no further discussion.