Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[benchmark] Extract Setup from Benchmarks #20048

Merged
merged 16 commits into from Nov 5, 2018

Commits on Oct 23, 2018

  1. Copy the full SHA
    4bbb635 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    8617745 View commit details
    Browse the repository at this point in the history
  3. [benchmark] Extract setup from ArrayAppend*

    ArrayAppendStrings had setup overhead of 10ms (42%). ArrayAppendLazyMap had setup overhead of 24 μs (1%).
    
    ArrayAppendOptionals and ArrayAppendArrayOfInt also had barely visible, small overhead of ~18μs, that was mostly hidden in measurement noise, but I’ve extracted the setup from all places that had 10 000 element array initializations, in preparation for more precise measurement in the future.
    palimondo committed Oct 23, 2018
    Copy the full SHA
    53653de View commit details
    Browse the repository at this point in the history
  4. [benchmark] Extract setup from ArrayInClass

    ArrayInClass had setup overhead of 88 μs (17%).
    palimondo committed Oct 23, 2018
    Copy the full SHA
    03d9841 View commit details
    Browse the repository at this point in the history
  5. [benchmark] Extract setup from DataBenchmarks

    DataCount had setup overhead of 18 μs (20%).
    DataSubscript had setup overhead of 18 μs (2%).
    SetUpFunction wasn’t necessary, because of short initialization (18 μs for `sampleData(.medium)`), which will inflate only the initial measurement.
    
    Runtimes of other benchmarks hide the sampleData initialization in their artificially high runtimes — most use internal multiplier of 10 000 iterations — but were changed to use the same constant data, since it was already available. The overhead will already be extracted if we go for more precise measurement with lower multipliers in the future.
    palimondo committed Oct 23, 2018
    Copy the full SHA
    ae9f5f1 View commit details
    Browse the repository at this point in the history
  6. [benchmark] Extract setup in Dictionary(OfObjects)

    Dictionary had setup overhad of 136 μs (6%).
    DictionaryOfObjects had setup overhead of 616 μs (7%).
    Also fixed variable naming convention (lowerCameCase).
    palimondo committed Oct 23, 2018
    Copy the full SHA
    32003a7 View commit details
    Browse the repository at this point in the history
  7. [benchmark] Extr. Setup DistinctClassFieldAccesses

    DistinctClassFieldAccesses had setup overhead of 4 μs (14%).
    Plus cosmetic code formatting fix.
    palimondo committed Oct 23, 2018
    Copy the full SHA
    63143d8 View commit details
    Browse the repository at this point in the history
  8. [benchmark] Extract setup from IterateData

    IterateData has setup overhead of 480 μs (10%).
    
    There remained strange setup overhead after extracting the data into setUpFunction, because of of-by-one error in the main loop. It should be either: `for _ 1…10*N` or: `for _ 0..<10*N`. It’s error to use 0…m*N, because this will result in `m*N + 1` iterations that will be divided by N in the reported measurement. The extra iteration then manifests as a mysterious setup overhead!
    palimondo committed Oct 23, 2018
    Copy the full SHA
    3ff92ef View commit details
    Browse the repository at this point in the history
  9. [benchmark] Extract setup from Phonebook

    Phonebook had setup overhead of 1266 μs (7%).
    palimondo committed Oct 23, 2018
    Copy the full SHA
    58a195f View commit details
    Browse the repository at this point in the history
  10. [benchmark] Extract setup from PolymorphicCalls

    PolymorphicCalls has setup overhead of 4 μs (7%).
    palimondo committed Oct 23, 2018
    Copy the full SHA
    4bc41f8 View commit details
    Browse the repository at this point in the history
  11. [benchmark] Fix setup overhead in RandomShuffle

    RandomShuffleLCG2 had setup overhead of 902 μs (17%) even though it already used the setUpFunction. Turns out that copying 100k element array is measurably costly.
    
    The only way to eliminate this overhead from measurement I could think of is to let the numbersLCG array linger around (800 kB), because shuffling the IOU version had different performance.
    palimondo committed Oct 23, 2018
    Copy the full SHA
    524de6b View commit details
    Browse the repository at this point in the history
  12. [benchmark] Extract setup from SortSortedStrings

    SortSortedStrings had setup overhead of 914 μs (30%).
    
    Renamed [String] constants to be shorter and more descriptive. Extracted the lazy initialiation of all these constants into `setUpFunction`, for cleaner measurements.
    palimondo committed Oct 23, 2018
    Copy the full SHA
    6d3e637 View commit details
    Browse the repository at this point in the history
  13. [benchmark] Extract setup from SubstringComparable

    SubstringComparable had setup overhead of 58 μs (26%).
    
    This was a tricky modification: extracting `substrings` and `comparison` constants out of the run function surprisingly resulted in decreased performance. For some reason this configuration causes significant increase in retain/release traffic. Aliasing the constants in the run function somehow works around this deoptimization.
    
    Also the initial split of the string into 8 substrings takes 44ms!!! (I’m suspecting some king of one-time ICU initialization?)
    palimondo committed Oct 23, 2018
    Copy the full SHA
    6b70314 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2018

  1. [benchmark] Extract setup: Sequence *Array benches

    Sequence benchmarks that test operations on Arrays have setup overhead of 14 μs. (Up from 4 μs a year ago!) That’s just the creation of an [Int] with 2k elements from a range… This array is now extracted into a constant.
    
    This commit also removes the .unstable tag from some CountableRange benchmarks, restoring them back to commit set of the Swift Benchmark Suite.
    palimondo committed Oct 24, 2018
    Copy the full SHA
    b55244d View commit details
    Browse the repository at this point in the history
  2. [benchmark] Extract setup: MapReduceClass(Small)

    MapReduceClass had setup overhead fo 868 μs (7%).
    
    Setup overhead of MapReduceClassShort was practically lost in the measurement noise from it’s artificially high base load, but it was there.
    
    Extracting the decimal array initialization into `SetUpFunction` also takes out the cost of releasing the [NSDecimalNumber], which turns out to be about half of the measured runtime in the case of MapReduceClass benchmark. This significantly changes the reported runtimes (to about half), therfore the modified benchmarks get a new name with suffix `2`.
    palimondo committed Oct 24, 2018
    Copy the full SHA
    96ff53d View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2018

  1. [benchmark] MapReduceClass2 and NSDecimalNumber

    Since this benchmark has been significantly modified and needs to be renamed, we can also lower the workload by a factor of 10, to keep up with the best practices.
    
    The old benchmark that uses `NSDecimalNumber` as the tested class is renamed to `MapReduceNSDecimalNumber` and the renamed `MapReduceClass2` now newly measures Swift class `Box` that wrap an `Int`. Short versions were modified analogously.
    palimondo committed Oct 30, 2018
    Copy the full SHA
    d8adfc7 View commit details
    Browse the repository at this point in the history