Misc. bug: Server multi-model mode add invalid `Transfer-Encoding: chunked` header to response

[EZForever]

Name and Version

Tested on latest build (b7240). Also tried #17698 but it does not seem to fix this issue.

C:\llama-b7240-bin-win-cpu-x64>llama-server.exe --version
load_backend: loaded RPC backend from C:\llama-b7240-bin-win-cpu-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llama-b7240-bin-win-cpu-x64\ggml-cpu-haswell.dll
version: 7240 (61bde8e21)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Linux, Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server.exe --models-dir "C:\models"

curl -v http://127.0.0.1:8080/completion -d '{"model":"stories260K","prompt":"hello","n_predict":10}'

Problem description & steps to reproduce

When running in multi-model mode, the proxy server add unnecessary and invalid Transfer-Encoding: chunked HTTP header to certain endpoints' responses, even if the mutually-exclusive Content-Length header is already present. It does not affect WebUI or any other ordinary usage, but will cause reverse proxies with strict header checking to fail (e.g. for Nginx it errors out on every completion request with upstream sent "Content-Length" and "Transfer-Encoding" headers at the same time while reading response header from upstream).

First Bad Commit

ec18edf, or since the introduction of multi-model mode feature.

Relevant log output

Single-model mode (--model C:\models\stories260K.gguf):

$ curl -v http://127.0.0.1:8080/completion -d '{"model":"stories260K","prompt":"hello","n_predict":10}'
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080
* using HTTP/1.x
> POST /completion HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/8.14.1
> Accept: */*
> Content-Length: 55
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 55 bytes
< HTTP/1.1 200 OK
< Server: llama.cpp
< Access-Control-Allow-Origin:
< Content-Type: application/json; charset=utf-8
< Content-Length: 1634
< Keep-Alive: timeout=5, max=100
<

Multi-model mode, b7240:

$ curl -v http://127.0.0.1:8080/completion -d '{"model":"stories260K","prompt":"hello","n_predict":10}'
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080
* using HTTP/1.x
> POST /completion HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/8.14.1
> Accept: */*
> Content-Length: 55
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 55 bytes
< HTTP/1.1 200 OK
< Server: llama.cpp
< Server: llama.cpp
< Access-Control-Allow-Origin:
< Access-Control-Allow-Origin:
< Connection: close
< Content-Length: 1630
< Content-Type: application/json; charset=utf-8
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Keep-Alive: timeout=5, max=100
<

Multi-model mode, #17698:

$ curl -v http://127.0.0.1:8080/completion -d '{"model":"stories260K","prompt":"hello","n_predict":10}'
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080
* using HTTP/1.x
> POST /completion HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/8.14.1
> Accept: */*
> Content-Length: 55
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 55 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: llama.cpp
< Access-Control-Allow-Origin:
< Connection: close
< Content-Length: 1636
< Content-Type: application/json; charset=utf-8
< Keep-Alive: timeout=5, max=100
<

[EZForever]

Followed a bunch of code and have found a possible cause:

On the processing of endpoint responses, a check is done to see if we should let httplib generate a chunked response:

llama.cpp/tools/server/server-http.cpp

Lines 337 to 338 in 37adc9c

	static void process_handler_response(server_http_res_ptr & response, httplib::Response & res) {
	if (response->is_stream()) {

llama.cpp/tools/server/server-http.cpp

Line 361 in 37adc9c

res.set_chunked_content_provider(content_type, chunked_content_provider, on_complete);

While the check is simply to look for the response generator function:

llama.cpp/tools/server/server-http.h

Lines 25 to 27 in 37adc9c

	bool is_stream() const {
	return next != nullptr;
	}

However, for a proxied request, the response is dynamicly read from a pipe, and thus the response generator function is always set regardless of the response itself is chunked or not, here in the constructor of server_http_proxy:

llama.cpp/tools/server/server-models.cpp

Lines 912 to 913 in 37adc9c

	// wire up the receive end of the pipe
	this->next = [pipe, should_stop](std::string & out) -> bool {

This causes process_handler_response to wrongly treat all proxied responses as chunked, and informs httplib as such. httplib then does its thing and adds Transfer-Encoding: chunked to HTTP headers, causing the problem:

llama.cpp/vendor/cpp-httplib/httplib.cpp

Lines 4276 to 4284 in 37adc9c

	void Response::set_chunked_content_provider(
	const std::string &content_type, ContentProviderWithoutLength provider,
	ContentProviderResourceReleaser resource_releaser) {
	set_header("Content-Type", content_type);
	content_length_ = 0;
	content_provider_ = detail::ContentProviderAdapter(std::move(provider));
	content_provider_resource_releaser_ = std::move(resource_releaser);
	is_chunked_content_provider_ = true;
	}

llama.cpp/vendor/cpp-httplib/httplib.cpp

Lines 5522 to 5524 in 37adc9c

	if (res.content_provider_) {
	if (res.is_chunked_content_provider_) {
	res.set_header("Transfer-Encoding", "chunked");

How this only affects some, not all of the endpoints remain a mystery. I don't currently have a viable fix on mind, so I hope the information above is helpful.

[arkamar]

I guess #17698 does not work for you because Access-Control-Allow-Origin: header is not listed in should_strip_proxy_header() function.

[arkamar]

I guess #17698 does not work for you because Access-Control-Allow-Origin: header is not listed in should_strip_proxy_header() function.

ok, probably no. The Control-Allow-Origin is no longer duplicated in my case.

[arkamar]

Anyway, this looks like a duplicate of #17693, which was just closed by #17698 merge.

[EZForever]

This is not a #17693 duplicate, and is not fixed by #17698. (In fact, the sample log provided by #17698's author also shows the exact same problem.) If your reverse proxy setup works with latest Nginx, then would you please share your Nginx config as a reference?

[arkamar]

Ah, yes, I see now, you are right. Actually, I am reproducing it in my nginx setup as well. I didn't notice this because streaming is always enabled in my workflow.

[arkamar]

cc: @ngxson, @ServeurpersoCom

[ngxson]

hmm yeah the proxy function always assume that we will stream the response. will see if we can have a quick fix for it

[ngxson]

upstream sent "Content-Length" and "Transfer-Encoding" headers at the same time while reading response header from upstream

In this case, does removing Content-Length works?

[ngxson]

Please have a look on #17734

[EZForever]

Can confirm that the fix works for Nginx.