Model generates only 16 Tokens always

[shrijayan]

Request body
{
"prompt": "\n\n### Instructions:\nwrite a story\n\n### Response:\n",
"stop": [
"\n",
"###"
]
}

Response Body
{ "id": "cmpl-f729eb01-6691-47c3-99f0-f30d6ab62f25",
"object": "text_completion", "created": 1699867559,
"model": "llama-2-13b-chat.Q4_0.gguf",
"choices": [
{ "text": "Once upon a time, in a far-off land, there was a mag",
"index": 0,
"logprobs": null,
"finish_reason": "length"
} ],

"usage":
{ "prompt_tokens": 20,
"completion_tokens": 16,
"total_tokens": 36
} }

In this, we can see only 16 completion_tokens generates all the time. How to solve this problem.

Terminal Input
python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1 --host 0.0.0.0 --port 8000 --n_ctx=2048 --n_ctx=2048

[tk-master]

--max_tokens ? idk why it would be this low by default though

[abetlen]

@tk-master this is to match the openai api defaults, but yes max_tokens will increase this and there's no default limit on the chat completion endpoint

[shrijayan]

@tk-master @abetlen

python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1 --host 0.0.0.0 --port 8000 --n_ctx=2048

This is the command that I used to start the FASTAPI, when I add --max_tokens=2048 it says main.py: error: unrecognized arguments: --max_tokens=2048.

[tk-master]

sorry, can you try something like..

{
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": [
    "\n",
    "###"
  ],
  "max_tokens" 200
}

[shrijayan]

sorry, can you try something like..

{
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": [
    "\n",
    "###"
  ],
  "max_tokens" 200
}

Hello @tk-master,

First of all thank you for answering the questions.

This is not working it gives the output like Internal Server Error. I also tried [max_tokens 200, "max_tokens"=200, max_tokens=200, n_ctx=200, "n_ctx"=200, "n_ctx" 200] all produces same output.

Without adding max_tokens is gives the "completion_tokens": 16.

[abetlen]

That should be "max_tokens": 200 otherwise the json is invalid. For example:

{
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": [
    "\n",
    "###"
  ],
  "max_tokens": 200
}

[shrijayan]

That should be "max_tokens": 200 otherwise the json is invalid. For example:

{
  "prompt": "\n\n### Instructions:\nWhat is the capital of France?\n\n### Response:\n",
  "stop": [
    "\n",
    "###"
  ],
  "max_tokens": 200
}

@abetlen Thank you so much this worked

[Mihaiii]

This is pretty silly. I also had this issue and I was confident it was a regression (everything worked fine before updating the library).

IMO this change should come with a bump major in version as it breaks current functionality.

[abetlen]

Hey @Mihaiii sorry for the trouble, yeah I should've avoided making that change. The intention was to match the OpenAI API defaults but obviously this was an unintentional breaking change and I'll avoid anything like that in the future. Cheers.

[segestic]

What worked for me.. -----------------------

Tried all above, it didn't work, number of tokens still remained same. What worked for me - Setting the stop to length of paragraphs e.g 3 paragraphs in below

{
  "prompt": "\n\n### Instructions:\n Write me a short poem \n\n### Response:\n",
  "stop": [
    "\n \n \n ",
    "###"
  ],
  "max_tokens": 200
}