[Bug][Spec Decode]: TPOT in prometheus is ITL in vllm serve

[ekagra-ranjan]

🐛 Describe the bug

There are 2 different kinds of metrics: ITL and TPOT. They are same for non SD model but are different for SD models. As per vLLM serve benchmark script, ITL measures the time to generate in 1 scheduler step whereas TPOT measures the time to generate 1 token.

However, the prometheus metric measures TPOT as this which corresponds to ITL in vLLM serve benchmark and contradicts with the behavior of TPOT. I think it should be renamed as ITL.

Also, is it possible to log the correct value for TPOT on prometheus so that one can measure the Output tokens/s reliably using prometheus?

cc: @markmc

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[dtransposed]

👍 This makes reading the Grafana dashboards super confusing, especially for users who are not aware of this quirk.