feat(benchmarks): Add Prefix Caching Benchmark to Serving Benchmark

[ywang96]

Reopens #3194 that was closed due to fork cleanup

Additional changes not mentioned in #3194 :

• Feature to save individual token latencies (itl) per request to the result json for further debugging
• Default vllm benchmark to OpenAI Completion API
• If streaming is supported, count output tokens by counting actual token streaming SSE events instead of tokenization will apply tokenization on output to count tokens since some API server does not stream token by token.
• Misc fixes/workaround in case the API server doesn't strictly follow Open API

To run the benchmark on the sonnet dataset, specify --dataset-name sonnet and --dataset-path <path to sonnet.txt>. Lengths of input, output and prefix lengths can be specified with command line args.

[ywang96]

@simon-mo Please take a look whenever you're free, thanks!

cc @robertgshaw2-neuralmagic in case you want to use this version of the server benchmark script - feel free to review it as well, thanks!

[ElizaWszola]

Would it make sense to, rather than getting dataset by name, specify whether we want to read from json or from a text file and provide a path to it? Or is it an overkill?

[ywang96]

Hmm - I think we can add the option to read from a json or path and that'll be nice for sure, but the problem is we'll then have to use a single user base prompt for all datasets. Also certain arguments of the benchmark (e.g., configure input, output & prefix length) is only used for certain dataset.

I haven't come up with a good solution so for now I keep them as separate datasets. I really like the idea of data registry where the way we sample prompts from the datasets & output lengths lives outside the main benchmark script, and we can reuse it for other benchmark scripts as well, but I haven't put too much thoughts into it

[ywang96]

@simon-mo I've addressed your comments as well as adding --dataset back so it's backward compatible for now with a deprecation warning. PTAL, thanks!

[dgisser]

@ywang96 is there any plan to post the metrics you've calculated based on these benchmark scripts (or the slides from the vllm meetup)?

[ywang96]

@ywang96 is there any plan to post the metrics you've calculated based on these benchmark scripts (or the slides from the vllm meetup)?

There is and stay tuned!