[benchmark] Janitor Duty, Legacy Factor: A-C #20861
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
palimondo
force-pushed
the
a-tall-white-fountain-played
branch
3 times, most recently
from
3decde6
to
ff688ab
Compare
|
@swift-ci please benchmark |
palimondo
changed the title
|
@palimondo Can you try again? The systems should now allow you to trigger swift-ci. |
|
@swift-ci please benchmark |
|
@swift-ci please test |
|
@palimondo Thanks! |
|
@shahmishal Oh, thank YOU! 🙏😊 |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
There are expected errors in @swift-ci please smoke test |
palimondo
changed the title
Lowered the default sample cap from 2k to 200. (This doesn’t effect manually specified `--num-samples` argument in the driver.) Swift benchmarks have pretty constant performance profile over time. It’s more beneficial to get multiple independent measurements faster, than more samples from the same run.
Reintroduce Ackermann benchmark with reasonably sized workload. Since this one was tagged `.unstable`, there’s no need to go through `legacyFactor`. Adjusted `lit` test for Benchmark_O now that Ackermann isn’t marked `.unstable` anymore. Removed incorrect `asserts` requirement from the benchmark lit tests.
Cleaned up doubled inner loops and applied legacyFactor to the resulting, more rational, workload sizes. Factors for string appending benchmarks are not 10, but 11 to compensate for smaller amortized Array resizes. Similarily ArrayPlusEqualFiveElementCollection’s factor is 49.
palimondo
force-pushed
the
a-tall-white-fountain-played
branch
from
23cb139
to
8fb9660
Compare
This comment has been minimized.
This comment has been minimized.
Lowered the workload of string appending bechmarks a bit further, to get them under 1ms in `O` and 10ms in `Onone` builds. Fine tuned the legacy factor to match original `O` runtimes — the changes are non-linear because of complications from double loop and amortized array growth costs. These string appending benchmarks along with * ArrayPlusEqualFiveElementCollection and * ArrayPlusEqualSingleElementCollection are likely to show different performance in `Osize` and `Onone` builds because of the non-linearity of their original implementation.
The `legacyFactor` is 11 instead of 10, because of the wierd way the `repeatCount` translates to the workload size based on the number of lines in the `workloadBase`… it’s just so.
Lowered the workload to get under 1ms runtime.
Also fixed variable workload size (`N`-dependent).
…plus an attempt at reducing wide range of memory used between independent, repeated measurements in the `CharacterPropertiesPrecomputed` benchmark by reserving the appropriate set capacity in advance.
palimondo
force-pushed
the
a-tall-white-fountain-played
branch
from
8fb9660
to
2121367
Compare
|
@swift-ci please benchmark |
|
@swift-ci please test |
|
@eeckstein Please review 🙏 |
palimondo
changed the title
Build comment file:Performance: -O
Code size: -O
Performance: -Osize
Code size: -Osize
Performance: -Onone
How to read the dataThe tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.If you see any unexpected regressions, you should consider fixing the Noise: Sometimes the performance results (not code size!) contain false Hardware Overview |
This was referenced Dec 18, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is first part of benchmarks clean-up to enable robust performance measurements by adjusting workloads to run in reasonable time (< 1000 μs), minimizing the accumulated error. To maintain long-term performance tracking, it applies legacy factor where necessary.
Legacy factor for benchmarks that had non-linear workload scaling (
ArrayAppend...) is tuned for-Obuilds, which means some changes in measured times for-Osizeand-Ononebuilds are unavoidable and to be expected, as a consequence of these workload scaling loops being fixed.