Eval bug: llama server response hangs for /slots/0?action=erase

[bmahabirbu]

Name and Version

b52edd2

Operating systems

Linux

GGML backends

CPU

Hardware

Mac M4

Models

llama3.2

Problem description & steps to reproduce

When running llama-server using ramalama (which runs llama.cpp inside the container) and with the necessary argument -slot-save-path /tmp to enable the slots feature when I try to do this command curl -X POST "http://localhost:8080/slots/0?action=erase" it will hang until i do control c then on the server side i see the response. But the response is never received by the curl command. I tried doing it inside the container as well to avoid networking issues but it still hangs

My goal is to clear the prompt cache for a summarization feature ie when the context size is reached clear the cache summarize the history and feed it back. The workaround is to just specify a small timeout but this seems like a bug.

ramalama latest llama.cpp commit = b52edd2

First Bad Commit

No response

Relevant log output

bmahabir@bmahabir-mac ramalama % curl -X POST "http://localhost:8080/slots/0?action=erase"
^C
bmahabir@bmahabir-mac ramalama % 


srv  remove_waiti: remove task 9 from waiting list. current waiting = 1 (before remove)
srv  log_server_r: request: POST /slots/0 192.168.127.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"id_slot":0,"n_erased":43}

The server log only happens after the control C. something is hanging in the llamaserver

[o7si]

Hi, @bmahabirbu, When executing curl -X POST "http://localhost:8080/slots/0?action=erase", the following code attempts to read content and gets stuck:

llama.cpp/vendor/cpp-httplib/httplib.h

Lines 7841 to 7844 in b52edd2

	if (!detail::read_content(strm, req, payload_max_length_, res.status, nullptr,
	out, true)) {
	return false;
	}

The issue is that the code tries to read content via detail::read_content(), which causes the request to hang.

You can use the following approaches to make the request work:

curl -X POST "http://localhost:8080/slots/0?action=erase" --data ""
# or
curl -X POST "http://localhost:8080/slots/0?action=erase" -H "Content-Length: 0"

[o7si]

The current implementation is based on cpp-httplib, but I'm not very familiar with the HTTP protocol and I'm not sure if this is the expected behavior.

I've submitted an issue (yhirose/cpp-httplib#2279) about this and hope someone with more expertise can help.

[bmahabirbu]

Thanks a bunch @o7si!! I will try this out

[o7si]

I noticed that the author has already implemented a fix upstream. I have submitted a related pull request (#17513).

[o7si]

Hi @bmahabirbu, the fix has been merged into the main branch (f3a848a). You can test based on the main branch using curl -X POST "http://localhost:8080/slots/0?action=erase" to observe if the stuck issue still occurs.