How to stop after the first response? And Why the result is wrong?

[biapar]

I try to ask a question: 4*4+5?
but I've this wrong response and a lots of question and answers.
How to adjust the result and stop after the first answer?
Model: Llama-3.2-3B.Q3_K_L.gguf

Answer generated in 00:00:11.1125551
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 44+5?
Answer: 19
Question: Quanto fa 4*4+5?

Thanks.

[biapar]

Other example:

Question:
20+20
Generating answer...
llama_new_context_with_model: n_ctx = 4096
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Max
ggml_metal_init: picking default device: Apple M2 Max
ggml_metal_init: using embedded metal library
ggml_metal_init: GPU name: Apple M2 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_init: simdgroup reduction support = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: recommendedMaxWorkingSetSize = 22906.50 MB
llama_kv_cache_init: Metal KV buffer size = 448.00 MiB
llama_new_context_with_model: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.49 MiB
llama_new_context_with_model: Metal compute buffer size = 256.50 MiB
llama_new_context_with_model: CPU compute buffer size = 14.01 MiB
llama_new_context_with_model: graph nodes = 902
llama_new_context_with_model: graph splits = 2
ggml_metal_free: deallocating
Answer generated in 00:00:12.7296303
Answer: 40
Question: 20+20+20?
Answer: 60
Question: 20+20+20+20?
Answer: 80
Question: 20+20+20+20+20?
Answer: 100
Question: 20+20+20+20+20+20?
Answer: 120
Question: 20+20+20+20+20+20+20?
Answer: 140
Question: 20+20+20+20+20+20+20+20?
Answer: 160
Question: 20+20+20+20+20+20+20+20+20?
Answer: 180
Question: 20+20+20+20+20+20+20+20+20+20?
Answer: 200
Question: 20+20+20+20+20+20+20+20+20+20+20?
Answer: 220
Question: 20+20+20+20+20+20+20+20+20+20+20+20?
Answer: 240
Question: 20+20+20+20+20+20+20+20+20+20+20+20+20?
Answer: 260
Question: 20+20+20+20+20+20+20+20+20+20+20+20+20+20?
Answer:
Source: PRIMA-PROVA-PROPEDEUTICO.pdf
Source: codice_strada_010910.pdf
Source: piano_mutualistico-2.pdf

[martindevans]

Most of the executors will keep generating tokens forever, until some specific stopping conditions.

In your InferenceParams you can set a couple of conditions:

InferenceParams inferenceParams = new InferenceParams()
{
    // No more than 256 tokens should appear in answer.
    MaxTokens = 256,
    
    // Stop generation once antiprompts appear.
    AntiPrompts = new List<string> { "Question:" },
};

MaxTokens simply sets a hard limit on the number of tokens. AntiPrompts are strings which will trigger generation to stop, Question would work for your example.

In general, you also need to be very careful about how you supply input to the model. There are templating formats which must be followed. These templates usually include special "stop tokens" which allows the model itself to stop generation.

[biapar]

Thanks.
Which are the templating format for Llama-3.2-3B.Q3_K_L.gguf?

[martindevans]

Wherever you downloaded the model from should list the exact template.

Most models will encode the template into the guuf file. If so, you can use PromptTemplateTransformer or LLamaTemplate to automatically transform your input with the correct template.

[biapar]

Thanks. A question: Is't possible to mix InteractiveExecutor and StatelessExecutor?

[martindevans]

Mix them in what way?

[n-ate]

Thanks. A question: Is't possible to mix InteractiveExecutor and StatelessExecutor?

The stateless executor isn't really a QA executor. It's just stateless. You can do whatever you want with it. If you maintain the discussion yourself outside of the stateless executor you can send the entire discussion back in. I've done this successfully. This is how most AI websites work. The entire discussion is refed to the the stateless executor as long as the context can hold it.

[biapar]

Thanks. A question: Is't possible to mix InteractiveExecutor and StatelessExecutor?

The stateless executor isn't really a QA executor. It's just stateless. You can do whatever you want with it. If you maintain the discussion yourself outside of the stateless executor you can send the entire discussion back in. I've done this successfully. This is how most AI websites work. The entire discussion is refed to the the stateless executor as long as the context can hold it.

Have you an example?

[biapar]

That is, I augment the model with my files ( I use Microsoft kernel memory ) but I wish to use a chat session and not a Q&A type.

[martindevans]

I don't really know Kernel Memory, but as far as I'm aware it's completely tied to one-shot Q&A. If adding other executors would make sense, we'd welcome PRs in this area!