[Usage]: What is the meaning of `Avg generation throughput`

[sleepwalker2017]

Your current environment

INFO 06-16 10:03:14 [loggers.py:87] Engine 000: Avg prompt throughput: 1254.4 tokens/s, Avg generation throughput: 29.2 tokens/s, Running: 10 reqs, Waiting: 4 reqs, GPU KV cache usage: 57.9%, Prefix cache hit rate: 40.5%

Does the Avg generation throughput means the tokens throughput?
Does it include the tokens generated by prefill ?

Can I use it as the metrics to measure the performance of throughput? Or it's only the decoding throughput ?

Thank you!

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[aarnphm]

Avg generation throughput is the number of generations tokens for this request. This doesn't include prefill token (prefill tokens are considered as prompt throughput here).

I think we ought to change this naming a bit

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!