[BUG] EOS token no longer triggering with KoboldCpp v1.34.2

[WolframRavenwolf]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

While benchmarking KoboldCpp v1.34.2 using the same setup (software, model, settings, deterministic preset, and prompts), the EOS token is not being triggered as with v1.33 anymore despite using --unbantokens. I expect the EOS token to be output and triggered consistently as it used to be with v1.33.

Current Behavior

I noticed generated text to be of lesser quality than it used to be with the previous version v1.33, more hallucinations and derailing the chat, and checked the console where I noticed it always triggered ending sequences. It used to trigger the EOS token, so I wonder if the lower quality output is caused by the missing EOS token, or is itself a symptom of some change that also causes the EOS token to be missing.

Environment and Context

Operating System: Windows 11
KoboldCpp: v1.34.2
Model: TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
Preset: "n": 1, "rep_pen": 1.18, "temperature": 0, "top_p": 0, "top_k": 1, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 2048, "rep_pen_slope": 0, "sampler_order": [0, 1, 2, 3, 4, 5, 6], "quiet": false, "max_context_length": 2048, "max_length": 250

I've been benchmarking various settings, but all exhibited the same behavior - here are the command lines:

• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --gpulayers 16 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --highpriority --unbantokens --usecublas --model TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --gpulayers 16 --highpriority --unbantokens --usecublas --model TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --gpulayers 16 --highpriority --unbantokens TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin
• koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --gpulayers 16 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin

The previous version properly triggered the EOS token - here is its command line:

• koboldcpp-1.33\koboldcpp.exe --blasbatchsize 1024 --gpulayers 16 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin

Failure Information (for bugs)

Although software, model, deterministic preset, and prompts were identical, clBLAS and cuBLAS generate very different outputs. Is that to be expected? But none of the runs produced the EOS token consistently, which worked very well with the version I used previously (v1.33). I can't say if the latest version's outputs are worse because of the missing EOS token, or if the EOS token is missing because the output is worse! I also don't know if the token is not generated by the model, suppressed by koboldcpp, or just ignored?

Steps to Reproduce

1. Use KoboldCpp v1.34.2 with --unbantokens and chat with a model that consistently produced them with previous versions.
2. Check console and notice instead of EOS token, it now triggers the ending sequence you definited, or goes on until max gen length.
3. Try the exact same model, settings and prompt with KoboldCpp v1.33 and you'll see the EOS token working.

Failure Logs

The only difference between the old and new version regarding logs is that now it's "Stop sequence triggered" instead of "EOS token triggered!"

[LostRuins]

I just tried with Wizard 7B on 1.34.2 with --unbantokens and had no issues triggering the EOS token. Is it possible that you're using too many stop tokens, or that your amount to generate is too low? Try going to instruct mode, removing all stop tokens, set amount to generate decently high, and ask a simple question.

Here is an example prompt: In one word, what is the meaning of life?

Input: {"n": 1, "max_context_length": 5555, "max_length": 32, "rep_pen": 1.08, "temperature": 0.7, "top_p": 0.92, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 1,
"rep_pen_range": 256, "rep_pen_slope": 0.7, "sampler_order": [6, 0, 1, 2, 3, 4, 5], "prompt": "\n### Instruction:\nIn one word, what is the meaning of life?\n###
Response:\n", "quiet": true, "stop_sequence": ["### Instruction:", "### Response:"]}

produced the expected output

Processing Prompt (17 / 17 tokens)
Generating (5 / 32 tokens)
(EOS token triggered!)
Time Taken - Processing:1.9s (110ms/T), Generation:0.3s (69ms/T), Total:2.2s (2.3T/s)
Output: Survival.

[LostRuins]

If it still doesn't work, please copy your console input and output for further debugging. You can also run it with --debugmode which will provide additional information.

[mrdevolver]

Hi, sorry for jumping in someone else's thread, but I think I have a similar problem. Yesterday, I was using guanaco-13b in Adventure. When I use Action, it always looks like '> I do this or that...', then the model tries to generate further development of the story and when it tries to make some actions on my behalf, it tries to write '> I...' but then the generation always stops. I couldn't figure out why it's happening, but I think I was also getting that 'Stop sequence triggered' in the console window, so I suspect it's due to the '>' character in the output? But I can't figure out how to prevent the model from starting with that character. I was trying to instruct the model to not use that character through memory and it just ignored my request, I also tried to use the new '--bantokens >' parameter, but then it didn't even start.

[WolframRavenwolf]

@LostRuins I've continued to investigate this issue and there's definitely something wrong here:

Using your example prompt "In one word, what is the meaning of life?" within the built-in Kobold Lite UI:

✔️ koboldcpp-1.33\koboldcpp.exe --debugmode --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_0.bin

Input:

Input: {"n": 1, "max_context_length": 2048, "max_length": 200, "rep_pen": 1.15, "temperature": 0.44, "top_p": 1, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 0.9, "rep_pen_range": 2048, "rep_pen_slope": 6.8, "sampler_order": [5, 0, 2, 3, 1, 4, 6], "prompt": "\n### Instruction:\nIn one word, what is the meaning of life?\n### Response:\n", "quiet": true, "stop_sequence": ["### Instruction:", "### Response:"]}

Output:

Processing Prompt (26 / 26 tokens)
Generating (1 / 200 tokens) [(Me 38.31%) (The 61.69%)]
Generating (2 / 200 tokens) [(an 100.00%)]
Generating (3 / 200 tokens) [(ing 100.00%)]
Generating (4 / 200 tokens) [(ful 100.00%)]
Generating (5 / 200 tokens) [(. 96.90%) ( 3.10%)]
Generating (6 / 200 tokens) [( 96.03%) (\n 3.97%)]

(EOS token triggered!)
Time Taken - Processing:24.9s (958ms/T), Generation:5.8s (970ms/T), Total:30.7s (0.2T/s)
Output: Meaningful.

❌ koboldcpp-1.34.2\koboldcpp.exe --debugmode --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_0.bin

Input:

Input: {"n": 1, "max_context_length": 2048, "max_length": 200, "rep_pen": 1.15, "temperature": 0.44, "top_p": 1, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 0.9, "rep_pen_range": 2048, "rep_pen_slope": 6.8, "sampler_order": [5, 0, 2, 3, 1, 4, 6], "prompt": "\n### Instruction:\nIn one word, what is the meaning of life?\n### Response:\n", "quiet": true, "stop_sequence": ["### Instruction:", "### Response:"]}

Output:

Processing Prompt (26 / 26 tokens)
Generating (1 / 200 tokens) [(The 100.00%)]
Generating (2 / 200 tokens) [( meaning 100.00%)]
Generating (3 / 200 tokens) [( of 100.00%)]
[...]
Generating (198 / 200 tokens) [( abs 100.00%)]
Generating (199 / 200 tokens) [(urd 100.00%)]
Generating (200 / 200 tokens) [(ity 100.00%)]

Time Taken - Processing:24.3s (935ms/T), Generation:230.1s (1151ms/T), Total:254.5s (0.8T/s)
Output: The meaning of life is 42.

This is a reference to Douglas Adams' "The Hitchhiker's Guide to the Galaxy" where the answer to the ultimate question of life, the universe, and everything is 42.

The number 42 was chosen by Adams as a joke and is not meant to have any particular significance. However, it has become a popular cultural reference and is often used to represent the search for meaning or the idea that the answer to life's questions may be found in unexpected places.

In the context of the story, the answer to the question of the meaning of life is revealed by a supercomputer named Deep Thought, which was designed to calculate the answer. After much deliberation, Deep Thought announces that the answer is 42, but also notes that the question itself was flawed and the real answer is meaningless.

Despite the absurdity

As you can see, the only difference between the two is the koboldcpp version, causing expectedly good output in the old version, but unexpectedly bad output in the new version.

I tested this with guanaco-33B.ggmlv3.q3_K_M.bin, guanaco-33B.ggmlv3.q4_0.bin, and guanaco-33B.ggmlv3.q4_K_M.bin, so I don't think it's caused by the quantization/format (which was done by the well-known TheBloke).

But I no longer think that it's an EOS issue - instead I think the missing EOS token is just a symptom of the model being brain-damaged when used with v1.34.2. This is shown better in a full chat using SillyTavern:

✔️ koboldcpp-1.33\koboldcpp.exe --blasbatchsize 1024 --debugmode --gpulayers 16 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin

❌ koboldcpp-1.34.2\koboldcpp.exe --blasbatchsize 1024 --debugmode --gpulayers 16 --highpriority --unbantokens --useclblast 0 0 TheBloke_guanaco-33B-GGML/guanaco-33B.ggmlv3.q4_K_M.bin

I've attached all the logs with --debugmode. I hope it helps you find out what's going on and find a fix.

• KoboldGPT Instruct-1.33.txt
• KoboldGPT Instruct-1.34.2.txt
• SillyTavern-1.33.txt
• SillyTavern-1.34.2.txt

[ycros]

@StefanDanielSchwarz this might be due to sampler_order now actually working, it used to be hardcoded at [6, 0, 1, 3, 4 ,2, 5] and you're sending [5, 0, 2, 3, 1, 4, 6].

I've put up a PR up to default back to the hardcoded order, putting the option to reorder behind a flag since the hardcoded order was probably optimal, but it's up to LostRuins as to what to do here.

For reference the numbering is this:

    KCPP_SAMPLER_TOP_K=0,
    KCPP_SAMPLER_TOP_A=1,
    KCPP_SAMPLER_TOP_P=2,
    KCPP_SAMPLER_TFS=3,
    KCPP_SAMPLER_TYP=4,
    KCPP_SAMPLER_TEMP=5,
    KCPP_SAMPLER_REP_PEN=6,

Try resorting to match the old order (rep pen, top k, top a, tfs, typical, top p, temp) in the SillyTavern UI and see if that helps.

[WolframRavenwolf]

@ycros Thank you very much! That seems to be exactly what's happening. I've changed the sampler order to [6, 0, 1, 3, 4, 2, 5] and now the responses are on point again and the EOS token is triggering again! 👍

@LostRuins Of course this is a bit of a problematic situation with no optimal solution as to what the default should be: Ignore sampler order unless a flag enables that feature, to keep backward compatibility and prevent unexpected breakage like that, or post a note about the change and ask people to adapt their sampler order. Same situation as with the EOS token being banned, I think that by default it should be unbanned and a flag should allow you to ban it if you want, but that's also a major change of the default. So in the end, if there's no optimal solution, I'd say go for consistency: Keep default behavior and add a flag for it like "unignore_sampler_order". Then for koboldcpp 2.0, you can bring the changed defaults with breaking changes. Just IMHO of course.

[LostRuins]

Yeah I'm a bit conflicted. I wanna try to avoid adding unnecessary flags, as that makes it more complicated for newbies, but I also want them to have a good default experience.

Alternative: The sampler orders are enabled by default, but using a suboptimal sampler order will display a warning in the console. @ycros @StefanDanielSchwarz I think that solution keeps flexibility for the pros and avoids complexity while steering newbies in the right direction.

[WolframRavenwolf]

I'm not sure if actual newbies know what sampler order is, how to change it, or even notice a warning message in the console. But having to enable a flag just so that a setting takes effect that should work out of the box is also not a good way to handle it.

So lacking a better alternative, I agree with your proposed one of enabling sampler orders by default (the way it should be) and displaying a warning for suboptimal sampler order. Now that I know what the issue is, I'll just adjust my presets.

Since I use SillyTavern with the proxy as my main setup, I'll see if I can make PRs to update their presets, or at least the defaults. Would make things even worse for newbies to have to update all of their presets themselves.

[LostRuins]

That's a good idea. I did have a talk with @RossAscends some time ago on optimal sampler order, so they are aware of it already and the defaults should be quite close, if not identical, to the optimal sampler order already.

[5, 0, 2, 3, 1, 4, 6] is an especially bad order though, since temperature should be last and rep pen should be first, and here they are swapped. Whatever preset this is, perhaps it should be revised.

[WolframRavenwolf]

I got the [5, 0, 2, 3, 1, 4, 6] from simple-proxy-for-tavern where it's used in a whole bunch of presets. Now that I know this order is problematic, I've gone through their presets and fixed them by taking SillyTavern's where both have the same preset or Kobold's recommended default. Made a PR and hope it will be accepted so it'll be fixed on that end. @anon998 FYI

SillyTavern's KoboldAI Settings presets look good to me, none of their sampler orders start with 5 or end with 6.

[LostRuins]

In the newest build, the console will now print a warning if bad sampler orders are detected.

[WolframRavenwolf]

Thank you! I'll close this issue.