Add benchmark serving to CI

[simon-mo]

• Currently we are downloading the full sharegpt but only sending 20 prompts, consider either smaller dataset or generate dummy inputs

[zhuohan123]

LGTM!

[simon-mo]

Working!

Serving Benchmarks
Namespace(backend='vllm', host='localhost', port=8000, endpoint='/v1/completions', model='meta-llama/Llama-2-7b-chat-hf', dataset='./ShareGPT_V3_unfiltered_cleaned_split.json', tokenizer='meta-llama/Llama-2-7b-chat-hf', best_of=1, use_beam_search=False, num_prompts=20, request_rate=inf, seed=0, trust_remote_code=False)

Total time: 58.91 s Throughput: 0.34 requests/s Average latency: 18.55 s Average latency per token: 0.05 s Average latency per output token: 0.10 s