Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-4600] Performance comparison should use MEAN and SD for analysis #47177

Open
palimondo mannequin opened this issue Apr 15, 2017 · 0 comments
Open

[SR-4600] Performance comparison should use MEAN and SD for analysis #47177

palimondo mannequin opened this issue Apr 15, 2017 · 0 comments
Assignees
Labels
bug A deviation from expected or documented behavior performance

Comments

@palimondo
Copy link
Mannequin

palimondo mannequin commented Apr 15, 2017

Previous ID SR-4600
Radar None
Original Reporter @palimondo
Type Bug
Status In Progress
Resolution
Additional Detail from JIRA
Votes 0
Component/s Project Infrastructure
Labels Bug, Performance
Assignee @palimondo
Priority Medium

md5: 006981c168e9f8486f59f9b3c2dddc48

is blocked by:

  • SR-4597 Benchmark results have wrong MEAN, MEDIAN and SD

Issue Description:

The compare_perf_tests.py performs statistically questionable analysis to determine what regressions and improvements are significant. This results in high noise in the results, forcing reviewers to perform more judgment calls than necessary.

In its current state, compare_perf_tests.py plays with MIN and MAX values to find some kind of significant performance change. But this is misguided. We are taking multiple samples of every performance test in order to eliminate one-off measurement aberrations (MIN, MAX) and getting to the truer value of MEAN. We have to use standard deviation (SD) to evaluate the difference between new and old values of MEAN represent meaningful improvements in performance of Swift.

To be fair, MEAN and SD values were probably ignored because they were incorrectly generated by Benchmark_Driver. That is SR-4597.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A deviation from expected or documented behavior performance
Projects
None yet
Development

No branches or pull requests

0 participants