Support OpenAI API server in benchmark_serving.py

[hmellor]

Adds --endpoint and --model parameters so that the user can benchmark their OpenAI compatible vLLM server

[hmellor]

@simon-mo can I request a review?

This might be something nice to merge before #2433 so that the OpenAI compatible server can benefit from @ywang96's work.

[simon-mo]

can you enable maintainers to edit this PR? I have some small fixes here

1. the tqdm part is wrong, it should use as_completed otherwise it is just measuring request sending, not completion
2. i want to add this to CI if possible

[hmellor]

can you enable maintainers to edit this PR?

Thanks for the review @simon-mo, unfortunately I don't think I can because my PR is from an organisation-owned fork, not a user-owned fork. Although, I'd be happy to make any changes you suggest!

[hmellor]

I have just realised that using tqdm.gather (as per my latest commit) is also not the right solution because it'll only get called to gather the last few tasks once the async for is complete.

[simon-mo]

Thanks for making the change. Let me add the benchmark script because it requires some back and forth.

[hmellor]

@simon-mo I've just confirmed this morning that the fix I made in ddd7e33 is not actually correct.

The progress bar only appears once the async for loop is complete so we only benefit from the progress bar once most of the requests have already completed.

[simon-mo]

Hmm I see. It is pretty tricky because a one liner won't do. Feel free to open another PR!

[tattrongvu]

Hi guys, thanks for the PR, just saw that in benchmark: https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py#L138

It doesn't count the tokens length in the output response, it take directly the max_tokens as output_len?

I think we should measure the actual output token throughput, not the requested max token.

[hmellor]

Hi @tattrongvu, this PR is only really concerned with enabling the benchmarking of the OpenAI compatible server, not with any specifics of the benchmark itself.

It doesn't count the tokens length in the output response, it take directly the max_tokens as output_len?

This is partially correct. You're right in saying that the actual output token length is not measured, but it's not using max_tokens either. From the code snippet below, you can see that the output_len actually comes from measuring the length of the completion in the dataset:

vllm/benchmarks/benchmark_serving.py

Lines 50 to 57 in 3209b49

	prompts = [prompt for prompt, _ in dataset]
	prompt_token_ids = tokenizer(prompts).input_ids
	completions = [completion for _, completion in dataset]
	completion_token_ids = tokenizer(completions).input_ids
	tokenized_dataset = []
	for i in range(len(dataset)):
	output_len = len(completion_token_ids[i])
	tokenized_dataset.append((prompts[i], prompt_token_ids[i], output_len))

This method is not great because it's very unlikely that any model will actually recreate the exact completion in the dataset.

The good news is that this is resolved in #2433!

[ywang96]

Hello @tattrongvu! As @hmellor mentioned, the original benchmark script always assumes the model generates the same number of tokens as output_len, which I believe won't be true for some engines (e.g., TGI) if they don't have a ignore_eos feature.

One of the fixes I did in #2433 is to postprocess the generated text and measure the number of tokens to give a more accurate result.