feat: cancel stream generation if client disappears

[tmm1]

Description

I added Context and Cancel fields to OpenAIRequest, and then ensured that the new per-request context was wired up to the grpc client.

Now if I ctrl+c my curl request while its streaming, the LocalAI server will show:

7:15PM DBG Sending chunk: {"object":"chat.completion.chunk","model":"llama2","choices":[{"index":0,"delta":{"content":" a"}
}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

7:15PM DBG Sending chunk failed: connection closed
Error rpc error: code = Canceled desc = context canceled

Notes for Reviewers

The next request after a cancellation is acting weird and produces some garbage tokens.

7:21PM DBG Sending chunk: {"object":"chat.completion.chunk","model":"llama2","choices":[{"index":0,"delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

7:21PM DBG Loading model llama from llama-2-13b-chat.ggmlv3.q4_K_M.bin
7:21PM DBG Model already loaded in memory: llama-2-13b-chat.ggmlv3.q4_K_M.bin
7:21PM DBG Model already loaded in memory: llama-2-13b-chat.ggmlv3.q4_K_M.bin
7:21PM DBG Sending chunk: {"object":"chat.completion.chunk","model":"llama2","choices":[{"index":0,"delta":{"content":" to"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Error rpc error: code = Unavailable desc = error reading from server: EOF

The third request will spin up the backend again and that works. Until another request is cancelled anyway.

Signed commits

• Yes, I signed my commits.

[mudler]

thanks! nice addition, just a small note from my side regarding the context. If we keep using the top level we are also sure that we can cancel it (particularly useful for tests)

[tmm1]

The next request after a cancellation is acting weird and produces some garbage tokens.

@mudler any thoughts on what might be causing this or how to go about fixing it?

[mudler]

The next request after a cancellation is acting weird and produces some garbage tokens.

@mudler any thoughts on what might be causing this or how to go about fixing it?

my guess is that as llama.cpp doesn't support multiple concurrent states - what probably happens is that the process is still busy inferencing, and another request tries to book the same context, failing