Perf: Analayze the roofline of the inference endpoints

[nvzhihanj]

We need to understand the roofline of:

• In offline, the maximum number of queries/reponses we can handle each second
• In online (concurrency), the maximum concurrency that we can measure for the endpoints
• In online, the maximum SSE chunks we can stream each second (which will impact our TPS roofline)

We can use SemiAnalsysis data as a reference: https://inferencemax.ai/

This will prepare us for the future when we need to horizontally scale to measure endpoints served on a larger cluster

[anandhu-eng]

Hi @nvzhihanj , I wanted to check whether anyone is currently working on this. If not, I would be interested in taking up the first part of the task.

Before starting, I’d like to clarify the expected scope. The issue mentions analyzing the roofline for offline QPS, online concurrency, and SSE chunk throughput. Since these metrics depend on the underlying system’s capabilities, I’m trying to understand whether the goal here is to design an approach or algorithm to identify the point at which the system begins to throttle or saturate and therefore determine when horizontal scaling is required?

[nvzhihanj]

Thanks @anandhu-eng , @viraatc was working a bit on this and can share if he had any progress on this.

You are correct that we need a way to identify the roofline of QPS/concrrency/SSE tput on the target system so the user is aware of the limit, and when to scale out.

[anandhu-eng]

Hey Zhihan, thanks for pointing it out. For starting, I was looking at the CLI args and think the following could have a direct impact on measuring the roofline:

--workers
--duration
--num-samples
--streaming
--min-output-tokens
--max-output-tokens
--load-pattern
--target-qps
--concurrency

[anandhu-eng]

Hey everyone, I’ve documented the various parameters that could impact roofline estimates and organized them into categories covering client-side factors, server-side factors, hardware constraints, and observability metrics. The goal is to serve as a starting point for systematic analysis and help us reason about which parameters to include when building heuristics or evaluating performance bottlenecks.

https://docs.google.com/document/d/1vDZ1EM_bDyy0bgG_9hGSfd9HJlT3iRksK4uhOVcFtq4/edit?usp=sharing

[viraatc]

update on this:

we have component tests available (see mr descriptions for latest perf):

• http-client roofline tests: feat: add perf utils #122
• zmq-transport roofline tests: feat: add zmq transport protocol #90

(1) In offline, the maximum number of queries/reponses we can handle each second
(2) In online (concurrency), the maximum concurrency that we can measure for the endpoints
(3) In online, the maximum SSE chunks we can stream each second (which will impact our TPS roofline)

(1) addressed in #122 (~200k QPS on x86, 150k QPS on x86 - 10 Cores; 1 Query = 1000 tokens)
(2) theoretical max. is ephemeral port limit (~22k) ; --max-concurrency rooflines depends on the Endpoint / OSL distribution
(3) addressed in #122 (--stream-interval=1 gives ~60k QPS on x86 - 108 Cores; 1 Query = 1000 tokens = 1000 SSE chunks)