Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Swift update performance benchmark infrastructure #7255

Merged
merged 3 commits into from
Apr 19, 2022
Merged

Swift update performance benchmark infrastructure #7255

merged 3 commits into from
Apr 19, 2022

Conversation

hassila
Copy link
Contributor

@hassila hassila commented Apr 19, 2022

Added dependency of Google's Swift benchmark library to the performance benchmark to provide more consistent results (it dynamically adjusts number of iterations depending on hardware) - also simplifies the benchmark as all supporting code could be removed as its provided for free by the package.

Sample output:

hassila@max ~/D/G/p/f/t/FlatBuffers.Benchmarks.swift (swift-update-performance-benchmark)> swift run -c release
Building for production...
Build complete! (0.07s)
running 10Strings... done! (1586.81 ms)
running 100Strings... done! (1668.91 ms)
running FlatBufferBuilder.add... done! (2332.52 ms)
running structs... done! (2263.61 ms)

name                  time            std        iterations
-----------------------------------------------------------
10Strings             14303124.500 ns ±   4.91 %         98
100Strings            28040333.000 ns ±   3.72 %         47
FlatBufferBuilder.add 82922708.000 ns ±   0.38 %         17
structs               80600250.000 ns ±   0.74 %         17

We also gain some tuning options:

hassila@max ~/D/G/p/f/t/FlatBuffers.Benchmarks.swift (swift-update-performance-benchmark)> .build/debug/FlatBuffers.Benchmarks.swift --help
USAGE: benchmark-command [--allow-debug-build] [--filter <filter>] [--filter-not <filter-not>] [--iterations <iterations>] [--warmup-iterations <warmup-iterations>] [--min-time <min-time>] [--max-iterations <max-iterations>] [--time-unit <time-unit>] [--inverse-time-unit <inverse-time-unit>] [--columns <columns>] [--format <format>] [--quiet]

OPTIONS:
  --allow-debug-build     Overrides check to verify optimized build.
  --filter <filter>       Run only benchmarks whose names match the regular expression.
  --filter-not <filter-not>
                          Exclude benchmarks whose names match the regular expression.
  --iterations <iterations>
                          Number of iterations to run.
  --warmup-iterations <warmup-iterations>
                          Number of warm-up iterations to run.
  --min-time <min-time>   Minimal time to run when automatically detecting number iterations.
  --max-iterations <max-iterations>
                          Maximum number of iterations to run when automatically detecting number iterations.
  --time-unit <time-unit> Time unit used to report the timing results.
  --inverse-time-unit <inverse-time-unit>
                          Inverse time unit used to report throughput results.
  --columns <columns>     Comma-separated list of column names to show.
  --format <format>       Output format (valid values are: json, csv, console, none).
  --quiet                 Only print final benchmark results.
  -h, --help              Show help information.

But also most importantly gives support for simple A/B performance comparisons:

hassila@max ~/D/G/p/f/t/FlatBuffers.Benchmarks.swift (swift-update-performance-benchmark) [64]> swift run -c release FlatBuffers.Benchmarks.swift --format json > a.json
Building for production...
Build complete! (0.07s)
running 10Strings... done! (1470.36 ms)
running 100Strings... done! (1706.32 ms)
running FlatBufferBuilder.add... done! (2325.53 ms)
running structs... done! (2249.58 ms)
hassila@max ~/D/G/p/f/t/FlatBuffers.Benchmarks.swift (swift-update-performance-benchmark)> swift run -c release FlatBuffers.Benchmarks.swift --format json > b.json
Building for production...
Build complete! (0.07s)
running 10Strings... done! (1518.49 ms)
running 100Strings... done! (1708.88 ms)
running FlatBufferBuilder.add... done! (2270.06 ms)
running structs... done! (2339.86 ms)
hassila@max ~/D/G/p/f/t/FlatBuffers.Benchmarks.swift (swift-update-performance-benchmark)> python3 ./compare.py a.json b.json
benchmark             column               a           b       %
----------------------------------------------------------------
10Strings             time       14333792.00 14212333.00    0.85
10Strings             std               4.74        3.67   22.48
10Strings             iterations       89.00       94.00   -5.62
100Strings            time       28220979.00 28556167.00   -1.19
100Strings            std               4.58        4.33    5.48
100Strings            iterations       48.00       48.00    0.00
FlatBufferBuilder.add time       82689750.00 82971666.50   -0.34
FlatBufferBuilder.add std               0.63        1.57 -150.56
FlatBufferBuilder.add iterations       17.00       16.00    5.88
structs               time       80078416.00 82826750.00   -3.43
structs               std               0.36        2.66 -629.54
structs               iterations       17.00       17.00    0.00
----------------------------------------------------------------
                      time                                 -1.02
                      std                                 -91.30
                      iterations                            0.15

(compare.py is here: https://github.com/google/swift-benchmark/blob/main/Scripts/compare.py )

hassila and others added 3 commits April 16, 2022 10:42
…der. Gives a significant performance boost for serialisation of many small messages.
…ynamic number of iterations, simplification of tests as result.
Copy link
Collaborator

@mustiikhalil mustiikhalil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mustiikhalil mustiikhalil merged commit 23c8ab3 into google:master Apr 19, 2022
@mustiikhalil
Copy link
Collaborator

@hassila Thanks for the contribution!

@hassila hassila deleted the swift-update-performance-benchmark branch April 19, 2022 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants