Integrate LightLLM into serve worker

[zeyugao]

Why are these changes needed?

Adapt from https://github.com/ModelTC/lightllm/blob/30304096f6f1ec050e2fac34b1795c9ee086ba63/lightllm/server/api_server.py

An alternative to vllm integration, in my own reproduce of https://github.com/ModelTC/lightllm/tree/30304096f6f1ec050e2fac34b1795c9ee086ba63#performance . In single A800 80g, LightLLM is about 20% faster than vllm (10.88 qps vs 8.95 qps). And I modify the benchmark script, directly benchmark the fastchat worker on two A800 80g, LightLLM is still faster than vllm (13.99 qps vs 12.42 qps).

However, the performance above is measure by api_server.py barely. After using the openai_api_server.py from fastchat. I observe some performance drop. Maybe due to the configuration.

It has some limitations, such as the lightllm does not provide the finish_reason. Awaiting ModelTC/lightllm#285 Done

Moreover, in the original implement of vllm integration, if we use /worker_generate and abort the connection, the vllm will not abort the engine, and it will continue try to generate the full content. In this pr, it shall be aware the disconnection of client and abort generation.

But if we use openai_api_server.py, even the client close the connection, due to that openai_api_server.py will not be notified about it, it will not close the connection between openai_api_server.py and model worker.

Related issue number (if applicable)

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

[merrymercy]

This looks great! Could you add some docs on how to use it?

[merrymercy]

Add a new documentation file or put some doc here on how to use this

[merrymercy]

@zeyugao any updates?

[zeyugao]

@zeyugao any updates?

Sorry for the late, I didn't receive the notifications when you place that request for change. And I am busy these days so I didn't check the pr status manually. I will fix that soon.

[merrymercy]

If you are interested, you can also try our new runtime https://github.com/sgl-project/sglang/tree/main and worker #2928, which we found often faster than vllm/lightllm

[zeyugao]

If you are interested, you can also try our new runtime https://github.com/sgl-project/sglang/tree/main and worker #2928, which we found often faster than vllm/lightllm

Thanks! I would take a look.

[zeyugao]

@merrymercy The doc is added