Bug: llama-service can only generate garbled text after a request with invalid tokens.

[morgen52]

What happened?

Hi there.

My llama-server initially worked fine, but after receiving a request with illegal characters, it started generating garbled responses to all valid requests.
For example, I start my llama-server with:

./llama.cpp-b3938/build_gpu/bin/llama-server     -m ../models/Meta-Llama-3-8B.Q4_K_M.gguf     -ngl 33  --cache-type-k q8_0

And I send a request with legal characters at first:

curl --request POST     --url http://localhost:8080/completion     --header "Content-Type: application/json"     --data '{         "prompt": "What is the meaning of life?\n\n",         "n_predict": 256 }' 

The response is as expected:

{"content":"The meaning of life is whatever you want it to be. But in a more serious sense, the meaning of life is to experience life, and to learn from experience.\n\nWhat is the most important thing in life?\n\nThe most important thing in life is love, because without love, life would have no meaning.","id_slot":0,"stop":true,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","tokens_predicted":63,"tokens_evaluated":8,"generation_settings":{"n_ctx":8192,"n_predict":-1,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","seed":4294967295,"seed_cur":1591500953,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"tfs_z":1.0,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"penalize_nl":false,"stop":[],"max_tokens":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"n_probs":0,"min_keep":0,"grammar":"","samplers":["top_k","tfs_z","typ_p","top_p","min_p","xtc","temperature"]},"prompt":"What is the meaning of life?\n\n","has_new_line":true,"truncated":false,"stopped_eos":true,"stopped_word":false,"stopped_limit":false,"stopping_word":"","tokens_cached":70,"timings":{"prompt_n":8,"prompt_ms":51.577,"prompt_per_token_ms":6.447125,"prompt_per_second":155.1078969308025,"predicted_n":63,"predicted_ms":1169.143,"predicted_per_token_ms":18.557825396825397,"predicted_per_second":53.88562391426883},"index":0}

And then I send a request with illegal characters:

curl --request POST     --url http://localhost:8080/completion     --header "Content-Type: application/json"     --data '{         "prompt": "What is the meaning of ÿþÿþÿþÿþÿþÿþÿþî?\n\n",         "n_predict": 256     }'

And I get a inconherent response, which is also as expected:

{"content":"GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG","id_slot":0,"stop":true,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","tokens_predicted":256,"tokens_evaluated":29,"generation_settings":{"n_ctx":8192,"n_predict":-1,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","seed":4294967295,"seed_cur":4216084932,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"tfs_z":1.0,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"penalize_nl":false,"stop":[],"max_tokens":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"n_probs":0,"min_keep":0,"grammar":"","samplers":["top_k","tfs_z","typ_p","top_p","min_p","xtc","temperature"]},"prompt":"What is the meaning of ÿþÿþÿþÿþÿþÿþÿþî?\n\n","has_new_line":false,"truncated":false,"stopped_eos":false,"stopped_word":false,"stopped_limit":true,"stopping_word":"","tokens_cached":284,"timings":{"prompt_n":29,"prompt_ms":53.786,"prompt_per_token_ms":1.854689655172414,"prompt_per_second":539.1737626891755,"predicted_n":256,"predicted_ms":4888.785,"predicted_per_token_ms":19.09681640625,"predicted_per_second":52.364749114555046},"index":0}

However, after that, I get garbled responses to all valid requests:

curl --request POST     --url http://localhost:8080/completion     --header "Content-Type: application/json"     --data '{         "prompt": "What is the meaning of life?\n\n",         "n_predict": 256     }'

And I still get the garbled response:

{"content":"GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG","id_slot":0,"stop":true,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","tokens_predicted":256,"tokens_evaluated":8,"generation_settings":{"n_ctx":8192,"n_predict":-1,"model":"../models/Meta-Llama-3-8B.Q4_K_M.gguf","seed":4294967295,"seed_cur":3234876351,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"tfs_z":1.0,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"penalize_nl":false,"stop":[],"max_tokens":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"n_probs":0,"min_keep":0,"grammar":"","samplers":["top_k","tfs_z","typ_p","top_p","min_p","xtc","temperature"]},"prompt":"What is the meaning of life?\n\n","has_new_line":false,"truncated":false,"stopped_eos":false,"stopped_word":false,"stopped_limit":true,"stopping_word":"","tokens_cached":263,"timings":{"prompt_n":8,"prompt_ms":51.324,"prompt_per_token_ms":6.4155,"prompt_per_second":155.8724962980282,"predicted_n":256,"predicted_ms":4874.52,"predicted_per_token_ms":19.04109375,"predicted_per_second":52.51799151506199},"index":0}

Name and Version

./llama.cpp-b3938/build_gpu/bin/llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
version: 7 (d9a33c5)
built with cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[slaren]

I cannot reproduce this, the response is coherent even with the "illegal" characters. I suggest trying a different model.

{
  "content": "I have an old MS-DOS text file that I want to load into a text file using a C# program. I have been using the File.ReadAllText function to load the text file but when I run the program, the file loads but it is full of characters like ÿþÿþÿþÿþÿþÿþÿþî and I have no clue what they are. I have tried changing the encoding from UTF8 to various other formats but the characters don't change.\n\n• Can you give an example of a single line from the file? – Jonathan Drapeau Nov 6 '18 at 23:18\n• I can't because of security reasons but I can tell you that every line starts with ÿþÿþÿþÿþÿþÿþÿþî and then goes on with a bunch of other characters. – user10211834 Nov 6 '18 at 23:23\n• OK, so the first character in each line is not a letter. – Jonathan Drapeau Nov 6 '18 at 23:25\n• I'm voting to close this question as off-topic because it is based on the premise that the",
  "id_slot": 0,
  "stop": true,
  "model": "models/meta-llama-3-8b.Q4_K_M.gguf",
  "tokens_predicted": 256,
  "tokens_evaluated": 29,
  "generation_settings": {
    "n_ctx": 8192,
    "n_predict": -1,
    "model": "models/meta-llama-3-8b.Q4_K_M.gguf",
    "seed": 4294967295,
    "seed_cur": 2877188899,
    "temperature": 0.800000011920929,
    "dynatemp_range": 0,
    "dynatemp_exponent": 1,
    "top_k": 40,
    "top_p": 0.949999988079071,
    "min_p": 0.05000000074505806,
    "xtc_probability": 0,
    "xtc_threshold": 0.10000000149011612,
    "tfs_z": 1,
    "typical_p": 1,
    "repeat_last_n": 64,
    "repeat_penalty": 1,
    "presence_penalty": 0,
    "frequency_penalty": 0,
    "dry_multiplier": 0,
    "dry_base": 1.75,
    "dry_allowed_length": 2,
    "dry_penalty_last_n": -1,
    "dry_sequence_breakers": [
      "\n",
      ":",
      "\"",
      "*"
    ],
    "mirostat": 0,
    "mirostat_tau": 5,
    "mirostat_eta": 0.10000000149011612,
    "penalize_nl": false,
    "stop": [],
    "max_tokens": 256,
    "n_keep": 0,
    "n_discard": 0,
    "ignore_eos": false,
    "stream": false,
    "n_probs": 0,
    "min_keep": 0,
    "grammar": "",
    "samplers": [
      "dry",
      "top_k",
      "tfs_z",
      "typ_p",
      "top_p",
      "min_p",
      "xtc",
      "temperature"
    ]
  },
  "prompt": "<|begin_of_text|>What is the meaning of ÿþÿþÿþÿþÿþÿþÿþî?\n\n",
  "has_new_line": true,
  "truncated": false,
  "stopped_eos": false,
  "stopped_word": false,
  "stopped_limit": true,
  "stopping_word": "",
  "tokens_cached": 284,
  "timings": {
    "prompt_n": 29,
    "prompt_ms": 243.204,
    "prompt_per_token_ms": 8.386344827586207,
    "prompt_per_second": 119.24145984441046,
    "predicted_n": 256,
    "predicted_ms": 2428.64,
    "predicted_per_token_ms": 9.486875,
    "predicted_per_second": 105.40878845773767
  },
  "index": 0
}

[morgen52]

Thanks for your response.
Do you set the --cache-type-k to precisions equal to or lower than q8_0? I find that this issue only appears when the ctk is set to q8_0 or less.

[slaren]

Yes, I used the same command line to start the server, and a model with the same file name I found on huggingface.

[morgen52]

I have tried the following steps:

1. Using the Phi-3-mini-4k-instruct-q4.gguf

./llama.cpp-b3938/build_gpu/bin/llama-server     -m ../models/Phi-3-mini-4k-instruct-q4.gguf    -ngl 33  --cache-type-k q8_0

I got

{"content":"################################################################################################################################################################################################################################################################","id_slot":0,"stop":true,"model":"../models/Phi-3-mini-4k-instruct-q4.gguf","tokens_predicted":256,"tokens_evaluated":10,"generation_settings":{"n_ctx":4096,"n_predict":-1,"model":"../models/Phi-3-mini-4k-instruct-q4.gguf","seed":4294967295,"seed_cur":3108310200,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"tfs_z":1.0,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"penalize_nl":false,"stop":[],"max_tokens":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"n_probs":0,"min_keep":0,"grammar":"","samplers":["top_k","tfs_z","typ_p","top_p","min_p","xtc","temperature"]},"prompt":"What is the meaning of life?\n\n","has_new_line":false,"truncated":false,"stopped_eos":false,"stopped_word":false,"stopped_limit":true,"stopping_word":"","tokens_cached":265,"timings":{"prompt_n":10,"prompt_ms":37.949,"prompt_per_token_ms":3.7948999999999997,"prompt_per_second":263.51155498168595,"predicted_n":256,"predicted_ms":3519.709,"predicted_per_token_ms":13.74886328125,"predicted_per_second":72.73328562105561},"index":0}

2. Using the Mistral-7B-Instruct-v0.3.Q4_1.gguf model

./llama.cpp-b3938/build_gpu/bin/llama-server     -m ../models/Mistral-7B-Instruct-v0.3.Q4_1.gguf    -ngl 33  --cache-type-k q8_0

I got an empty content.

{"content":"","id_slot":0,"stop":true,"model":"../models/Mistral-7B-Instruct-v0.3.Q4_1.gguf","tokens_predicted":256,"tokens_evaluated":10,"generation_settings":{"n_ctx":32768,"n_predict":-1,"model":"../models/Mistral-7B-Instruct-v0.3.Q4_1.gguf","seed":4294967295,"seed_cur":3003397237,"temperature":0.800000011920929,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"tfs_z":1.0,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"penalize_nl":false,"stop":[],"max_tokens":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"n_probs":0,"min_keep":0,"grammar":"","samplers":["top_k","tfs_z","typ_p","top_p","min_p","xtc","temperature"]},"prompt":"What is the meaning of life?\n\n","has_new_line":false,"truncated":false,"stopped_eos":false,"stopped_word":false,"stopped_limit":true,"stopping_word":"","tokens_cached":265,"timings":{"prompt_n":10,"prompt_ms":47.068,"prompt_per_token_ms":4.706799999999999,"prompt_per_second":212.45857057873718,"predicted_n":256,"predicted_ms":4720.973,"predicted_per_token_ms":18.44130078125,"predicted_per_second":54.22610974474965},"index":0}

3. Using the Meta-Llama-3-8B-Instruct.Q8_0.gguf model

../llama.cpp/build_gpu/bin/llama-server     -m ../models/Meta-Llama-3-8B-Instruct.Q8_0.gguf     -ngl 33  --cache-type-k q8_0

I got the same behavior as reported above, i.e., unexpected output after the first request with 'invalid' tokens.

4. Update the llama.cpp to the newest released version, b3985. The behavior becomes normal.

It seems to have been fixed in the latest version. Thank you for your help.