Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

put the compare perf script into the repo #564

Merged
merged 1 commit into from
Aug 10, 2018
Merged

put the compare perf script into the repo #564

merged 1 commit into from
Aug 10, 2018

Conversation

weissi
Copy link
Member

@weissi weissi commented Aug 9, 2018 •
edited
Loading

Motivation:

For a long while I had a script which runs NIO's performance tests
against different Swift toolchains. It should be in the repository.

Modifications:

add scripts/compare_perf_of_swift_versions.sh

Result:

everybody can run the performance tests against different Swift versions

Example output:

benchmark:  bytebuffer_lots_of_rw 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.990602 0.99001 0.9784911 0.979174 0.9497989 0.9619211 0.971509 1.107166 1.001857 0.9670759 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.9498  0.9682  0.9788  0.9898  0.9905  1.1072 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.692134 0.688547 0.717389 0.698619 0.685348 0.6760701 0.686126 0.70909 0.6847429 0.6896141 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.6761  0.6855  0.6891  0.6928  0.6970  0.7174 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -28.82 %

benchmark:  bytebuffer_write_http_response_ascii_only_as_string 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.287226 0.302724 0.296216 0.2937721 0.2949389 0.3264929 0.300645 0.290748 0.2972779 0.2986599 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2872  0.2941  0.2967  0.2989  0.3001  0.3265 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.079831 0.07798004 0.07679605 0.08038902 0.07931399 0.07545698 0.08269203 0.08085489 0.08236611 0.08195698 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.07546 0.07831 0.08011 0.07976 0.08168 0.08269 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -73.73 %

benchmark:  bytebuffer_write_http_response_ascii_only_as_staticstring 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.03858101 0.03899992 0.03870499 0.04407609 0.03999996 0.039994 0.04033804 0.05134904 0.06055009 0.04102099 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.03858 0.03925 0.04017 0.04336 0.04331 0.06055 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.03645897 0.03486907 0.03389001 0.0340569 0.03387308 0.03363502 0.03420699 0.03385401 0.03357399 0.03536999 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.03357 0.03386 0.03397 0.03438 0.03470 0.03646 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -12.98 %

benchmark:  bytebuffer_write_http_response_some_nonascii_as_string 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.733603 0.7239901 0.716985 0.7277981 0.739315 0.7360909 0.729737 0.735379 0.7359639 0.745093 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.7170  0.7283  0.7345  0.7324  0.7361  0.7451 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.847316 0.866539 0.885741 0.863849 0.8311881 0.8454369 0.830847 0.8314171 0.835175 0.822601 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.8226  0.8312  0.8403  0.8460  0.8597  0.8857 
winner:  /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : 14.73 %

benchmark:  bytebuffer_write_http_response_some_nonascii_as_staticstring 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.04189706 0.04078901 0.04370904 0.04293203 0.04090905 0.04129398 0.04043996 0.04029894 0.04050493 0.04121101 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.04030 0.04058 0.04106 0.04140 0.04175 0.04371 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.03429699 0.03392696 0.03407907 0.03534102 0.03388095 0.03393006 0.03523397 0.03583503 0.03438497 0.03382301 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.03382 0.03393 0.03419 0.03447 0.03502 0.03584 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -16.07 %

benchmark:  no-net_http1_10k_reqs_1_conn 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.249231 0.24253 0.252389 0.252063 0.247653 0.247869 0.250036 0.253286 0.247592 0.2477211 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2425  0.2477  0.2485  0.2490  0.2516  0.2533 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.228726 0.229262 0.230268 0.2325931 0.232911 0.2329639 0.23089 0.2267009 0.224956 0.2272921 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.2250  0.2277  0.2298  0.2297  0.2322  0.2330 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -7.25 %

benchmark:  http1_10k_reqs_1_conn 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.6686051 0.661356 0.685306 0.6713231 0.66308 0.6936001 0.674819 0.713164 0.691676 0.698909 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.6614  0.6693  0.6801  0.6822  0.6931  0.7132 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.6950361 0.6890179 0.6893619 0.6752219 0.6746031 0.6856769 0.678369 0.691776 0.671314 0.67413 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.6713  0.6748  0.6820  0.6825  0.6893  0.6950 
winner:  /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : 1.51 %

benchmark:  http1_10k_reqs_100_conns 
===========
- with /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift 
0.82987 0.892851 1.070676 0.9005131 0.859078 0.832703 0.7907679 0.878788 0.7725201 0.799472 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.7725  0.8071  0.8459  0.8627  0.8893  1.0707 
- with /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
0.7263739 0.7311001 0.724852 0.732116 0.735194 0.770691 0.759672 0.742476 0.7611519 0.7595961 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.7249  0.7314  0.7388  0.7443  0.7597  0.7707 
winner:  /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift 
comparison: /Library/Developer/Toolchains/swift-4.1-RELEASE.xctoolchain/usr/bin/swift to /Library/Developer/Toolchains/swift-4.2-DEVELOPMENT-SNAPSHOT-2018-08-07-a.xctoolchain/usr/bin/swift : -6.17 %

@weissi
put the compare perf script into the repo
9d78ab8 
Motivation:

For a long while I had a script which runs NIO's performance tests
against different Swift toolchains. It should be in the repository.

Modifications:

add scripts/compare_perf_of_swift_versions.sh

Result:

everybody can run the performance tests against different Swift versions
@weissi weissi requested a review from Lukasa August 9, 2018 17:33
@Lukasa Lukasa added the semver/patch No public API change. label Aug 10, 2018
@Lukasa Lukasa added this to the 1.10.0 milestone Aug 10, 2018
@Lukasa Lukasa merged commit 8078cc8 into apple:master Aug 10, 2018
@weissi weissi deleted the jw-compare-perf-script branch August 10, 2018 13:41
@weissi weissi modified the milestones: 1.10.0, 1.9.3 Aug 28, 2018
weissi added a commit that referenced this pull request Aug 29, 2018
@weissi
put the compare perf script into the repo (#564)
10354f3 
Motivation:

For a long while I had a script which runs NIO's performance tests
against different Swift toolchains. It should be in the repository.

Modifications:

add scripts/compare_perf_of_swift_versions.sh

Result:

everybody can run the performance tests against different Swift versions
Motivation:

Explain here the context, and why you're making that change.
What is the problem you're trying to solve.

Modifications:

Describe the modifications you've done.

Result:

After your change, what will change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Lukasa Lukasa Lukasa approved these changes

Assignees
No one assigned
Labels
semver/patch No public API change.
Projects
None yet
Milestone
1.9.3
Development

Successfully merging this pull request may close these issues.
2 participants
@weissi @Lukasa