server : return HTTP 400 if prompt exceeds context length

[rgerganov]

In streaming mode when prompt exceeds context length, the server returns HTTP 200 status code with a JSON error in the body. This is very confusing and inconsistent with all other inference engines which return HTTP 4xx error in this case.

This patch fixes this problem and makes the server return HTTP 400 in such cases.

[ngxson]

Hmm that's strange, we have a specific error type for this, ERROR_TYPE_EXCEED_CONTEXT_SIZE. The error code is 400:

llama.cpp/tools/server/server.cpp

Lines 1268 to 1271 in 56b4795

	case ERROR_TYPE_EXCEED_CONTEXT_SIZE:
	type_str = "exceed_context_size_error";
	code = 400;
	break;

We also have this test case:

llama.cpp/tools/server/tests/unit/test_chat_completion.py

Lines 393 to 408 in 56b4795

	def test_context_size_exceeded():
	global server
	server.start()
	res = server.make_request("POST", "/chat/completions", data={
	"messages": [
	{"role": "system", "content": "Book"},
	{"role": "user", "content": "What is the best book"},
	] * 100, # make the prompt too long
	})
	assert res.status_code == 400
	assert "error" in res.body
	assert res.body["error"]["type"] == "exceed_context_size_error"
	assert res.body["error"]["n_prompt_tokens"] > 0
	assert server.n_ctx is not None
	assert server.n_slots is not None
	assert res.body["error"]["n_ctx"] == server.n_ctx // server.n_slots

I'm wondering which input leads to the 200 code that you mentioned?

[rgerganov]

The issue occurs only in streaming mode. In non-streaming it correctly returns 400.

[rgerganov]

I have added a new test which covers exceeding the context in streaming mode.

[ngxson]

I'm thinking maybe this check can better be done inside launch_slot_with_task? There you will have access to slot.n_ctx

[rgerganov]

Unfortunately there is no way to return non-200 status code once you call res.set_chunked_content_provider(...). That's why I am doing the check before that.

[ngxson]

Hmm ok then, I'll refactor this code in #16488 , for now this can be a temporary soltuion

[ngxson]

If this is handled inside launch_slot_with_task, you can call send_error(slot, ".....", ERROR_TYPE_EXCEED_CONTEXT_SIZE);, which should simplify things a bit