Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Changes at 4/19/2022 9:35:37 PM #4817

Closed
performanceautofiler bot opened this issue Apr 26, 2022 · 2 comments
Closed

[Perf] Changes at 4/19/2022 9:35:37 PM #4817

performanceautofiler bot opened this issue Apr 26, 2022 · 2 comments

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 4.56 ms 8.34 ms 1.83 0.08 True
Count - Duration of single invocation 3.31 ms 6.74 ms 2.04 0.07 True
Count - Duration of single invocation 4.51 ms 8.29 ms 1.84 0.06 True

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[^\n]*", Options: NonBacktracking)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.340815666666668 > 4.726036502497475.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -57.2571338950626 (T) = (0 -8390487.941977082) / Math.Sqrt((7818886183.785275 / (23)) + (148372360025.30594 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.8579772658824698 = (4515926.053590281 - 8390487.941977082) / 4515926.053590281 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?s).*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.738108863523573 > 3.4824430852604173.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -230.11277573533923 (T) = (0 -6760860.29769661) / Math.Sqrt((4163092571.808942 / (23)) + (1425926982.3002477 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -1.027713587578754 = (3334228.433005471 - 6760860.29769661) / 3334228.433005471 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: ".*", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.285871557692309 > 4.731868686555944.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -72.49287773610047 (T) = (0 -8355328.40275375) / Math.Sqrt((1522638542.8598952 / (23)) + (96277890522.02982 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.8535516993334773 = (4507739.603787831 - 8355328.40275375) / 4507739.603787831 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Threading.Channels.Tests.BoundedChannelPerfTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
PingPong - Duration of single invocation 10.68 ms 11.48 ms 1.08 0.12 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Threading.Channels.Tests.BoundedChannelPerfTests*'

Payloads

Baseline
Compare

Histogram

System.Threading.Channels.Tests.BoundedChannelPerfTests.PingPong


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.483906828054298 > 11.177547596358696.
IsChangePoint: Marked as a change because one of 3/4/2022 11:33:07 AM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -13.454632532900792 (T) = (0 -11457107.121401245) / Math.Sqrt((51819395511.96745 / (23)) + (78626422250.46277 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.0855089872305328 = (10554594.44019146 - 11457107.121401245) / 10554594.44019146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SingleSerial - Duration of single invocation 320.73 ns 377.65 ns 1.18 0.11 False
SingleSerial - Duration of single invocation 304.77 ns 363.47 ns 1.19 0.18 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Byte>.SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 377.64991607142855 > 337.06842618571426.
IsChangePoint: Marked as a change because one of 2/23/2022 3:28:40 AM, 4/8/2022 10:19:59 AM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -30.00621472037777 (T) = (0 -375.3227486811438) / Math.Sqrt((39.06110740213733 / (23)) + (36.12035619091378 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.15221060159970629 = (325.74144705842247 - 375.3227486811438) / 325.74144705842247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Byte&gt;.SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 363.46922040816327 > 322.73015333571436.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -25.614525912904604 (T) = (0 -364.7889477103773) / Math.Sqrt((65.04840047003277 / (23)) + (54.09452180995144 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.17212572413881427 = (311.2199828038033 - 364.7889477103773) / 311.2199828038033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Threading.Channels.Tests.UnboundedChannelPerfTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
PingPong - Duration of single invocation 11.24 ms 12.22 ms 1.09 0.08 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Threading.Channels.Tests.UnboundedChannelPerfTests*'

Payloads

Baseline
Compare

Histogram

System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.221611837301584 > 11.713151503260871.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 4/8/2022 7:02:49 PM, 4/13/2022 1:59:07 PM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -16.9173345628046 (T) = (0 -12121900.066502059) / Math.Sqrt((26176971242.8094 / (23)) + (84054772863.84938 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.09055028050787453 = (11115397.687905625 - 12121900.066502059) / 11115397.687905625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Threading.Tasks.Tests.Perf_AsyncMethods

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Yield - Duration of single invocation 286.02 ns 335.84 ns 1.17 0.07 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Threading.Tasks.Tests.Perf_AsyncMethods*'

Payloads

Baseline
Compare

Histogram

System.Threading.Tasks.Tests.Perf_AsyncMethods.Yield


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 335.8411737333334 > 298.34728274749995.
IsChangePoint: Marked as a change because one of 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -36.33611859898587 (T) = (0 -335.5713782329692) / Math.Sqrt((24.539556620704968 / (23)) + (30.74939443819776 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.1779003623284728 = (284.88944308465267 - 335.5713782329692) / 284.88944308465267 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline f1e10a53132cc648cea914e75f953ff5a5087399
Compare de0b85f2b1690355f05bf13d3a88b8828b9853b1
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SingleSerial - Duration of single invocation 345.52 ns 385.09 ns 1.11 0.12 False
SingleSerial - Duration of single invocation 341.15 ns 384.71 ns 1.13 0.05 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: True)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 385.0914007857143 > 357.5385098666667.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -21.419046122139918 (T) = (0 -381.2374003492809) / Math.Sqrt((76.94792665571985 / (23)) + (34.07846515728208 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.13219986600310113 = (336.72270400024644 - 381.2374003492809) / 336.72270400024644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;.SingleSerial(RentalSize: 4096, ManipulateArray: False, Async: True, UseSharedPool: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 384.7131822222222 > 359.7747919428571.
IsChangePoint: Marked as a change because one of 3/4/2022 11:33:07 AM, 4/8/2022 10:19:59 AM, 4/19/2022 4:39:14 PM, 4/26/2022 6:18:08 AM falls between 4/17/2022 3:25:40 PM and 4/26/2022 6:18:08 AM.
IsRegressionStdDev: Marked as regression because -20.754947919742296 (T) = (0 -381.1052267487689) / Math.Sqrt((45.19552534822438 / (23)) + (39.32008869080374 / (35))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (35) - 2, .025) and -0.1058372434970825 = (344.630486077289 - 381.1052267487689) / 344.630486077289 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@AndyAyersMS
Copy link
Member

Nominal range is dotnet/runtime@4881a63...eb9dd67

For regexp tests this is likely dotnet/runtime#68199

For other tests perhaps dotnet/runtime#68171 ?

@dakersnar
Copy link

dakersnar commented Jun 8, 2022

Regressions for System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong detected on Arm64 Linux and x86 Windows from the perf report. Historical data for each pictured below, respectively. I'm diagnosing this as Noise, but wanted to add this data here for when we Look Again. Seems to be correlated to Alloc Delta (see details).

image

image

System.Threading.Channels.Tests.UnboundedChannelPerfTests.PingPong

Result Ratio Alloc Delta Operating System Bit Processor Name
Slower 0.59 +153 debian 11 Arm64 Unknown processor
Same 0.98 -240 ubuntu 18.04 Arm64 Unknown processor
Slower 0.53 +153 ubuntu 20.04 Arm64 Unknown processor
Same 0.96 -7 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.02 +15 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 0.95 -21 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 0.94 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Same 0.98 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Same 1.02 -8 Windows 11 X64 AMD Ryzen 9 5900X
Same 1.02 -21 Windows 11 X64 AMD Ryzen 9 5950X
Same 1.01 +0 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Same 1.01 -4 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Same 0.95 +0 ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
Same 1.02 +18 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Same 1.02 +0 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.15 +461 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 0.93 +8 macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants