Making /v1beta/chat/completions streaming output compatible with openai

[wsxiaoys]

Please describe the feature you want

For now /v1beta/chat/completions generate streaming outputs as below (streamming json lines):

{"content":" In"}
{"content":" Python"}
{"content":","}
{"content":" you"}
{"content":" can"}
{"content":" convert"}
{"content":" a"}
{"content":" list"}
{"content":" of"}
{"content":" strings"}
{"content":" to"}
{"content":" numbers"}
{"content":" using"}
{"content":" the"}
{"content":" `"}
{"content":"map"}
{"content":"()"}
{"content":"`"}
{"content":" function"}
{"content":" and"}
{"content":" the"}
{"content":" `"}
{"content":"int"}
{"content":"()"}
{"content":"`"}
{"content":" function"}
{"content":"."}
{"content":" Here"}
{"content":"'"}
{"content":"s"}
{"content":" an"}
{"content":" example"}
{"content":":"}

We'd like to make the response format compatible with openai's text/event-stream streaming response.

Additional context

Discuss in slack: https://tabbyml.slack.com/archives/C05CWLZ0Y85/p1701451409878009

Code Location: https://github.com/TabbyML/tabby/blob/main/crates/tabby/src/routes/chat.rs#L39

llama.cpp's server example on text/event-stream: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/server.cpp#L2775

[heurainbow]

Should consider following common cases (within a max output token limit):
1 max generate some lines, say 3
2 complete a function with some clear return signal w.r.t language
3 special tokens
Better to directly support vLLM backend in #795

[brian316]

can we setup openai chat with tabby? per docs doesnt show how https://tabby.tabbyml.com/docs/administration/model/#chat-model

[wsxiaoys]

Hi - the http backend can be configured in following way to use openai chat

[model.chat.http]
kind = "openai/chat"
model_name = "<model name>"
api_endpoint = "https://api.openai.com/v1"
api_key = "secret-api-key"