Benchmark overhaul #41408
Merged
Benchmark overhaul #41408
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
McPatate
approved these changes
Oct 13, 2025
Member
McPatate
left a comment
McPatate
left a comment
There was a problem hiding this comment.
btw to disable the associated CI workflows while I rework them later:
on:
workflow_dispatch:and you can delete the rest of the triggers
A few naming comments overall, but minor stuff I believe, feel free to ignore, gj 👌🏻
remi-or
force-pushed
the
benchmark-various
branch
from
c7e15c3 to
d2c89aa
Compare
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
McPatate
reviewed
Oct 14, 2025
i3hz
pushed a commit
to i3hz/transformers
that referenced
this pull request
Oct 15, 2025
* Big refactor, still classes to move around and script to re-complexify * Move to streamer, isolate benches, propagate num tokens * Some refacto * Added compile mode to name * Re-order * Move to dt_tokens * Better format * Fix and disable use_cache by default * Fixed compile and SDPA backend default * Refactor results format * Added default compile mode * Always use cache * Fixed cache and added flex * Plan for missing modules * Experiments: no cg and shuffle * Disable compile for FA * Remove wall time, add sweep mode, get git commit * Review compliance, start * Apply suggestions from code review Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Update benchmark_v2/framework/benchmark_runner.py Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Disable workflow * Pretty print * Added some pretty names to have pretty logs * Review n2 compliance (end?) * Style and end of PR --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
ngazagna-qc
pushed a commit
to ngazagna-qc/transformers
that referenced
this pull request
Oct 23, 2025
* Big refactor, still classes to move around and script to re-complexify * Move to streamer, isolate benches, propagate num tokens * Some refacto * Added compile mode to name * Re-order * Move to dt_tokens * Better format * Fix and disable use_cache by default * Fixed compile and SDPA backend default * Refactor results format * Added default compile mode * Always use cache * Fixed cache and added flex * Plan for missing modules * Experiments: no cg and shuffle * Disable compile for FA * Remove wall time, add sweep mode, get git commit * Review compliance, start * Apply suggestions from code review Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Update benchmark_v2/framework/benchmark_runner.py Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Disable workflow * Pretty print * Added some pretty names to have pretty logs * Review n2 compliance (end?) * Style and end of PR --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR overhauls the benchmarking suite that is included in transformers.
The benchmarking suite is now based around three main components:
BenchmarkingConfigis a dataclass-like object which contains everything needed to reproduce a benchmark on the same machine: input length, generation length, whether to usekernelsorcompile, attention implementation, etc. (subject to name change)BenchmarkRunneris the class that runs the benchmarks defined by the configs, with a given number of measurement iterations, warmup iterations, and a model-id. The runner takes care of setting up the runs in a way that ensures no run interacts with the downstream ones: the model is reloaded, the cache is emptied and the GPU memory is flushed. It also saves the results, the config, and any additional metadata needed to reproduce the benchmark, like hardware information and package versions.For now, the new benchmarking suite replaces the
benchmark_v2part oftransformersbut it could also overwrite thebenchmark(v1) part. It would be good to make that decision in this PR. And update the CI workflows that rely on the currentbenchmark_v2(putting the PR in draft mode until then).An example of how to use the new benchmarking suite can be found in
run_benchmarks.py.The format of the results file can (and may be bound to) change as we develop tools to analyze them.
If there is a metric you want to see measured in
transformers, please leave a comment before this is merged 🙂