[RFC][libc++][benchmark] Enable benchmark optimizations. #132445
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before we moved our benchmark suite from a stand-alone suite to the current lit based suite the default was to run benchmarks optimized. With the change the default is to run not optimized. I've been bitten several times by this issue and need to spend time to look how to run these test optimized.
I feel this is a regression and also makes it harder for other people to do the right thing. I feel the default should be to do the right thing. I agree this somewhat differs from the idea that the test suite should be run with the same flags. Still I feel having the right default makes it easier to onboard in the project and not to remember these oddities.
This patch adds a new default optimization (bikeshedding is welcome) that keeps the main tests as if specified optimization=none and the benchmarks as if specified optimization=speed.
The other values keep their exisiting behaviour. (When the test are optimized for speed or size the benchmarks will inherit this flag without any changes.) The benchmarks can still be not opimized by using optimization=none.
This patch will need more cleanups and documentation.