-
|
When benchmarking vLLM on a single GPU, I want a cost-per-token figure that reflects idle time between requests, not just the active inference window. Do you amortize the full pod uptime (hourly GPU price × wall-clock) over total tokens served, or only count tokens during active batches? The two give very different numbers under bursty traffic. What approach did you settle on in this lab? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
After testing both, this lab amortizes full pod wall-clock uptime over total tokens served, not just active-batch time. Rationale: under real serving you pay for the GPU whether or not a request is in flight, so €/token = (hourly_price × uptime_hours) / total_tokens. Counting only active windows flatters the number and hides the cost of idle capacity — which is exactly the thing FinOps needs to see under bursty load. The practical mitigation is scale-to-zero (teardown when idle) so uptime tracks demand; the runbooks measure both the raw amortized figure and an ideal fully-utilized floor, and report the gap as the utilization tax. |
Beta Was this translation helpful? Give feedback.
After testing both, this lab amortizes full pod wall-clock uptime over total tokens served, not just active-batch time.
Rationale: under real serving you pay for the GPU whether or not a request is in flight, so €/token = (hourly_price × uptime_hours) / total_tokens. Counting only active windows flatters the number and hides the cost of idle capacity — which is exactly the thing FinOps needs to see under bursty load.
The practical mitigation is scale-to-zero (teardown when idle) so uptime tracks demand; the runbooks measure both the raw amortized figure and an ideal fully-utilized floor, and report the gap as the utilization tax.