Disable Early Stopping for LLM benchmarks

[attafosu]

Early stopping (ES) criteria as implemented for the inference benchmarks provides robust determination for test systems to meet stipulated latency constraints, particularly for the Server scenario. While this purpose is very well served in general benchmarks, the same cannot be said about the LLM benchmarks. For instance, submitters often may have a difficult time during performance, especially even when the SUT meets the stipulated token latency constraints.
For the LLM benchmarks, there’re few things that make ES difficult to use:

1. Equal-issue mode is enabled by default: Unlike other benchmarks where the performance samples (subset of the dataset that is used for performance runs) is a smaller subset of the validation, LLM benchmarks use the full validation set for the run. This ensures that all samples in the dataset are seen at least once during the benchmark run, and that latency results are not skewed.
   The dataset also tends to be a carefully curated representation of the samples typically found in the task for the benchmark, as well as the SLAs. As such, at the end of a benchmark run, the latency profile of the SUT is a reliable performance indicator on the given task/dataset.

2. As stated previously, tuning the server QPS has become challenging: Given the two latency constraints for LLM benchmarks, TTFT and TPOT, users have to spend time tuning their QPS, which may meet latency constraints, but barely passes ES. Combined with point 1 above, if a system passes the two latency constraints, it should be reasonable to conclude of its performance thereof.

3. Moreover, benchmark duration for LLMs typically goes well beyond the minimum duration requirement of 10min. ES was initially proposed to replace the minimum query count, so that systems that comfortably meet the latency constraints can terminate early.
   “We propose replacing the minimum query count with a function that dynamically determines when a run has reached statistical significance. Systems that consistently deliver low latency queries will have shorter run times; systems that struggle to hit the latency bound may have longer run times.
   Specifically, implementations with underlatency percentiles greater than the target tail latency will have much lower required query counts and therefore runtime. Conversely, implementations with underlatency percentiles that are very close to the target tail latency percentile will have somewhat higher minimum query counts. The net effect will be shorter runtimes for small systems in the SingleStream, MultiStream, and Server scenarios”

Given that LLM benchmarks can’t “terminate” early even if the SUT comfortably meets the latency constraints during the run, enforcing ES for these benchmarks makes no sense.
For these reasons, we propose to make the ES criteria optional for LLM benchmarks.

[nvzhihanj]

@nv-jinhosuh to help review

[nv-jinhosuh]

Thanks @attafosu for the proposal. I am hesitant using ES for any workload that has nonuniform distribution, but at least it is theoretically proved to be valid for Server scenarios. And we all agreed that using whole dataset and randomize the whole dataset multiple times (the more the better) can bring the statistics working behind to healthy state.

So even if we disable ES, we still need the whole dataset (equal issue mode) for any workload we see non-uniform workload. And the question of how many samples we need to run, still be very high (would be much higher than ES) since the whole dataset would be considered as 'one representative sample' from statistics point of view. When we apply this to the original paper calculating number of required samples to run assuming workload processing latency follows Gaussian distribution, for specific confidence interval, we will need to run tens of thousands times the whole dataset (while we randomize samples within each run of total dataset). So I think ES is actually helping us to handle this for very slow, low throughput workloads like LLMs.

Now, we use metrics like token based latency and throughputs. Doesn't this solve the non-uniform sample problem? I am not confident. Even though we set max ISL/OSL, the token latency distribution would differ based on each sample's ISL. Unfortunately I didn't/couldn't follow all the discussions around this. Do you have any scatter plot or histogram showing output token latency distribution per input in the LLM workloads MLPerf Inference uses? If we see they are pretty uniform, and can say workload is nonuniform in our latency response, only based on OSL, then I think we can reconsider how many samples we need to run, and should we run whole dataset with equal-issue-mode, given we remove 'response latency averaged for all the request samples' (that is we only care for TTFT and TPOT, and TPS, and remove RPS from our metrics together with LoadGen Early Stopping estimates statistics based on tokens not requests - would need change in the LoadGen).

[attafosu]

Hi @nv-jinhosuh On the issue of equal issue mode, yes, the LLM benchmarks are required to use the whole dataset mainly due to the non-homogeneity of the input lengths. See attached for dataset used for Llama3-405B:

The distribution of the first token latency (TTFT) also have correlation with the ISL and thus will be non-uniform as well (don't have the plots yet).
Just to clarify: Is your argument that if we should get rid of ES then the min_query requirement prior to ES introduction should be reinstated? If so, I believe that estimation also uses a margin of error that may not reflect the level of variation in the token latencies, especially the TTFT, whether measured with fixed arrival interval (no loadgen) or poison arrival rate (with loadgen).

[nv-jinhosuh]

Thank you @attafosu - right, this is what I am afraid of; current LLM performance is dependent on many different things even in token granularity, so all the statistics we relied on (Gaussian distribution of 'uniform' response characteristics due to jitter) simply don't work well. ISL dependency is one thing, even if we somehow give unlimited memory capacity, there will be other parameters to chime in to change the token level characteristics further.

So what I am trying to raise is, ES is actually helping us, as it is providing macro level (sample request and sample response) statistics (with the help from equal issue mode) confidence into our runs. When we remove ES, we will need to come up with certain 'bar' i.e. minimum number of requests, and that would likely be 'many many' times of total input dataset size. For example, before ES is there, we had to run 24K samples for 90%-ile (https://arxiv.org/abs/1911.02549 TABLE IV) tail latency. I think the equivalent of this is 24K * total dataset size, in our case. With ES, we may be okay running 10 * total dataset size and the ES would estimate acceptable statistic behavior and let the run pass (This kind of making sense as running whole dataset one time being one sample, you would expect extremely small deviation/jitter between sample to sample). The multiplier may end up being smaller if we somehow can capture the parameters around the bell curve though, but I think that would be still larger (pessimistic being safer) than what we could use in ES.

I cannot think of any other statistical basis to decide minimum number of samples to run to make the benchmarking scientifically healthy though.

Eventually, we may want someone with strong statistics background to revise the 'requirement' here. I think we have been touching this non-uniform sample problem from 5 years ago, and found partial solutions - relying on the law of large number. But given the LLM is low throughput, with nonuniform response characteristic, this becomes a challenging issue.

[arjunsuresh]

@nv-jinhosuh Is 99%-ile tail latency still relevant for LLMs? If this can be reduced I believe ES requirements will be correspondingly reduced.

[nv-jinhosuh]

@arjunsuresh No idea what target metric is relevant for LLM but there should be minimum bar. Might be a good topic to bring up in the LLM WG.

I guess the point though is, ES will help even if we relax it to something we think minimal here as a hardline basis (min_query_count = XXX). Parameters we use for ES would also could be reduced as you suggested; that doesn't mean our run will automatically reduced in runtime as ES won't do dynamic runtime adjustment (no halt when it sees it passes anytime in the middle; technically we can enable it though, accepting it wouldn't know if next 1000 samples would cause violating the estimation, but that also applies to our ES management through min_query_count though).

I think the easier, feasible solution is to random sample the dataset to come up with 'small/reduced dataset' like CPU SPEC Benchmark does, if runtime is a general concern... And the submission may happen in full_dataset separately in column vs reduced_dataset, understanding reduced_dataset has its own implications...

[attafosu]

@nv-jinhosuh @arjunsuresh @nvzhihanj @nv-ananjappa @mrmhodak Trying to summarize the discussion so far:

• Early stopping is even more necessary for LLM benchmarks - which tend to have longer run times
• If we still want to get rid of early stopping, we need to have strict requirement on the minimum query count. While this count may not be as high as the previous 270k for server (margin of error parameter used in that estimate may have been too low for current llm benchmarks), it still would be high enough to render must run times impractical
• What are the possibility and implications of adjusting the tail latency percentile (downwards/upwards) from 99%?

These concerns/questions need to be thoroughly addressed before pulling the plug on Early Stopping for llms. The downside still is submitters contending with painful performance tuning, but until we figure out resolutions to the above concerns, it's a necessary "evil"

[nv-jinhosuh]

Yes - thank you @attafosu for the nice summary.

My intuition says, if one's throughput is low, one can tune the parameters while running 'one' set of full dataset, i.e equal-issue-mode=on, min_query_count=dataset_size, min_duration=1
Due to the (large) size of the dataset, any first time ramp up or final ramp down would impact minimally (likely shadowed behind the long inference time, making the deviation jitter very low).
Once the tuning is done (maybe little pessimistic first), one can run longer run to see how it goes.

[mrmhodak]

This proposal is being dropped.