New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[benchmark] Extract Setup from Benchmarks #20048
Commits on Oct 23, 2018
-
-
-
[benchmark] Extract setup from ArrayAppend*
ArrayAppendStrings had setup overhead of 10ms (42%). ArrayAppendLazyMap had setup overhead of 24 μs (1%). ArrayAppendOptionals and ArrayAppendArrayOfInt also had barely visible, small overhead of ~18μs, that was mostly hidden in measurement noise, but I’ve extracted the setup from all places that had 10 000 element array initializations, in preparation for more precise measurement in the future.
-
[benchmark] Extract setup from ArrayInClass
ArrayInClass had setup overhead of 88 μs (17%).
-
[benchmark] Extract setup from DataBenchmarks
DataCount had setup overhead of 18 μs (20%). DataSubscript had setup overhead of 18 μs (2%). SetUpFunction wasn’t necessary, because of short initialization (18 μs for `sampleData(.medium)`), which will inflate only the initial measurement. Runtimes of other benchmarks hide the sampleData initialization in their artificially high runtimes — most use internal multiplier of 10 000 iterations — but were changed to use the same constant data, since it was already available. The overhead will already be extracted if we go for more precise measurement with lower multipliers in the future.
-
[benchmark] Extract setup in Dictionary(OfObjects)
Dictionary had setup overhad of 136 μs (6%). DictionaryOfObjects had setup overhead of 616 μs (7%). Also fixed variable naming convention (lowerCameCase).
-
[benchmark] Extr. Setup DistinctClassFieldAccesses
DistinctClassFieldAccesses had setup overhead of 4 μs (14%). Plus cosmetic code formatting fix.
-
[benchmark] Extract setup from IterateData
IterateData has setup overhead of 480 μs (10%). There remained strange setup overhead after extracting the data into setUpFunction, because of of-by-one error in the main loop. It should be either: `for _ 1…10*N` or: `for _ 0..<10*N`. It’s error to use 0…m*N, because this will result in `m*N + 1` iterations that will be divided by N in the reported measurement. The extra iteration then manifests as a mysterious setup overhead!
-
[benchmark] Extract setup from Phonebook
Phonebook had setup overhead of 1266 μs (7%).
-
[benchmark] Extract setup from PolymorphicCalls
PolymorphicCalls has setup overhead of 4 μs (7%).
-
[benchmark] Fix setup overhead in RandomShuffle
RandomShuffleLCG2 had setup overhead of 902 μs (17%) even though it already used the setUpFunction. Turns out that copying 100k element array is measurably costly. The only way to eliminate this overhead from measurement I could think of is to let the numbersLCG array linger around (800 kB), because shuffling the IOU version had different performance.
-
[benchmark] Extract setup from SortSortedStrings
SortSortedStrings had setup overhead of 914 μs (30%). Renamed [String] constants to be shorter and more descriptive. Extracted the lazy initialiation of all these constants into `setUpFunction`, for cleaner measurements.
-
[benchmark] Extract setup from SubstringComparable
SubstringComparable had setup overhead of 58 μs (26%). This was a tricky modification: extracting `substrings` and `comparison` constants out of the run function surprisingly resulted in decreased performance. For some reason this configuration causes significant increase in retain/release traffic. Aliasing the constants in the run function somehow works around this deoptimization. Also the initial split of the string into 8 substrings takes 44ms!!! (I’m suspecting some king of one-time ICU initialization?)
Commits on Oct 24, 2018
-
[benchmark] Extract setup: Sequence *Array benches
Sequence benchmarks that test operations on Arrays have setup overhead of 14 μs. (Up from 4 μs a year ago!) That’s just the creation of an [Int] with 2k elements from a range… This array is now extracted into a constant. This commit also removes the .unstable tag from some CountableRange benchmarks, restoring them back to commit set of the Swift Benchmark Suite.
-
[benchmark] Extract setup: MapReduceClass(Small)
MapReduceClass had setup overhead fo 868 μs (7%). Setup overhead of MapReduceClassShort was practically lost in the measurement noise from it’s artificially high base load, but it was there. Extracting the decimal array initialization into `SetUpFunction` also takes out the cost of releasing the [NSDecimalNumber], which turns out to be about half of the measured runtime in the case of MapReduceClass benchmark. This significantly changes the reported runtimes (to about half), therfore the modified benchmarks get a new name with suffix `2`.
Commits on Oct 30, 2018
-
[benchmark] MapReduceClass2 and NSDecimalNumber
Since this benchmark has been significantly modified and needs to be renamed, we can also lower the workload by a factor of 10, to keep up with the best practices. The old benchmark that uses `NSDecimalNumber` as the tested class is renamed to `MapReduceNSDecimalNumber` and the renamed `MapReduceClass2` now newly measures Swift class `Box` that wrap an `Int`. Short versions were modified analogously.