.NET 7.0 Preview 5 Microbenchmarks Performance Study Report #70590
Labels
area-Meta
tenet-performance
Performance related issue
tracking
This issue is tracking the completion of other related issues.
Milestone
Comments
dakersnar
added
area-Meta
tenet-performance
ghost
added
the
|
Tagging subscribers to this area: @dotnet/area-meta Issue DetailsDataThis time we have covered following configs:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL. This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report! The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them! Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here. Notable Improvements
RegressionsBy design
Investigation in progress
Already solved
Noise, flaky or multimodalThe following benchmarks showed up in the report generated by the tool, but were not actual regressions:
StatisticsTotal: 81617 Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
|
|
Am I reading this correctly? |
|
@TonyValenti That's correct. One of the purposes of this report is to provide an additional line of defense against unintended regressions that can slip past us. We analyze the data and try to track down what might be causing specific regressions. Sometimes, the reported regression ends up just being a noisy benchmark. Sometimes, there is an actual regression, and the report helps notify the right people about the problem, while providing data to help them solve it.
If you happen to be looking at the full report generated by the tool when you come to this conclusion, I want to clarify that the data is sorted from most regressed to most improved. Only looking at the top of the file will lead to that conclusion, but the whole report is 100k lines long, and most benchmarks in the middle end up not changing on a month-to-month basis. Let me know if that cleared things up! |
|
Also worth noting that some regressions are from a bug fix (#68691). If you factor those out it won't look quite as bad. |
|
There was also a particularly high number of noisy benchmarks this month, a lot of which were incorrectly flagged as regressions by the tool. This unfortunately skews the overall statistics. You can see specifics in my |
On top of what others said, # tests does not necessarily correlate well with # distinct regressions, # apps affected, etc. often a regression in many tests is caused by one or two regressions, which can then get fixed. Also, this is a point in time during development -- the key thing is the system spots regressions in time to fix them. |
24 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Data
This time we have covered following configs:
Most of the benchmarks were run on bare-metal machines, some were executed via WSL.
This would not be possible without the help from: @AndyAyersMS @carlossanlop @janvorli @jeffhandley @sblom and @tannergooding who contributed their results and time. An addtional thank you to @mrsharm for shadowing the creation of this report!
The full report generated by the tool is available here. You will have to click "Raw" to see the entire file. The report is sorted from most regressed to most improved, so scroll to the bottom in the full report to see improvements. There are plenty of them!
Again, the full historical data turned out to be extremely useful. For details about methodology please read #41871. Preview 4 report can be found here.
Notable Improvements
System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE,System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBESystem.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{2,4}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled),System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: ".{0,2}(Tom|Sawyer|Huckleberry|Finn)", Options: Compiled)PerfLabTests.CastingPerf.IFooFooIsIFoo,PerfLabTests.CastingPerf.ObjFooIsObj2,PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFooSystem.Memory.Span<Byte>.Reverse(Size: 512),System.Memory.Span<Char>.Reverse(Size: 512),System.Memory.Span<Int32>.Reverse(Size: 512),System.Tests.Perf_Array.ReverseSystem.Memory.Span<Byte>.Fill(Size: 512),System.Memory.Span<Byte>.Clear(Size: 512)Regressions
By design
System.Numerics.Tests.Perf_Quaternion.EqualsBenchmark,System.Numerics.Tests.Perf_Vector3.EqualsBenchmark,System.Numerics.Tests.Perf_Vector2.EqualsBenchmark,System.Numerics.Tests.Perf_Matrix3x2.EqualsBenchmark,System.Numerics.Tests.Perf_Vector4.EqualsBenchmarkInvestigation in progress
System.Text.Perf_Utf8Encoding.GetString(Input: Chinese),System.Text.Perf_Utf8Encoding.GetString(Input: Greek),System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic),System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)char.Is*update #69727char.Is*update #69727 (comment)System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, False)),System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, False)),System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, False)),System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False)),System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreNonSpace, False)),System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, False))System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))System.Tests.Perf_String.Format_MultipleArgsSystem.Collections.Sort<IntStruct>.List(Size: 512),System.Collections.Sort<IntStruct>.Array(Size: 512)System.Memory.Span<Byte>.Reverse(Size: 512),System.Memory.Span<Char>.Reverse(Size: 512),System.Memory.Span<Int32>.Reverse(Size: 512),System.Tests.Perf_Array.ReverseAlready solved
MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_,MicroBenchmarks.Serializers.Json_FromStream<Location>.DataContractJsonSerializer_,MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_,MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>.DataContractSerializer_,MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.DataContractSerializer_, MicroBenchmarks.Serializers.Xml_FromStream.DataContractSerializer_`Benchstone.BenchI.IniArray.TestNoise, flaky or multimodal
The following benchmarks showed up in the report generated by the tool, but were not actual regressions:
System.Text.RegularExpressions.Tests.*.Count(*, Options: None)Options: Noneregex tests, but historical data indicates it is probably noise. These are worth keeping an eye on.BenchmarksGame.RegexRedux_5.RunBench(options: None)System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPongSystem.Tests.Perf_Char.Char_ToLowerInvariant(input: "Hello World!"),System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)System.Net.Security.Tests.SslStreamTests.ConcurrentReadWriteSystem.Numerics.Tests.Perf_Matrix4x4.IdentityBenchmarkPerfLabTests.CastingPerf.CheckListIsVariantGenericInterfaceSystem.Buffers.Tests.RentReturnArrayPoolTests<Byte>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)System.Numerics.Tests.Perf_Vector3.DistanceSquaredBenchmarkSystem.Numerics.Tests.Perf_Vector2.ReflectBenchmarkSystem.Numerics.Tests.Perf_VectorOf<Double>.NegateBenchmarkSystem.Numerics.Tests.Perf_VectorOf<Int32>.XorBenchmarkPerfLabTests.LowLevelPerf.GenericGenericMethodPerfLabTests.LowLevelPerf.GenericClassWithSTringGenericInstanceMethodSystem.Memory.Span<Int32>.IndexOfAnyFourValues(Size: 512)System.Collections.IterateForEach<Int32>.LinkedList(Size: 512)System.Memory.Span<Byte>.SequenceCompareTo(Size: 512)Statistics
Total: 81617
Same: 79.67 %
Slower: 7.73 %
Faster: 4.27 %
Noise: 8.27 %
Unknown: 0.05 %
Statistics per Architecture
Statistics per Operating System
Statistics per Namespace
Big thanks to everyone involved!
The text was updated successfully, but these errors were encountered: