Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/arm64: 28 Regressions on 7/18/2023 1:13:00 AM #89259

Closed
performanceautofiler bot opened this issue Jul 20, 2023 · 7 comments
Closed

[Perf] Windows/arm64: 28 Regressions on 7/18/2023 1:13:00 AM #89259

performanceautofiler bot opened this issue Jul 20, 2023 · 7 comments
Labels
arch-arm64 area-Infrastructure os-windows PGO runtime-coreclr specific to the CoreCLR runtime tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
GetBytes - Duration of single invocation
📝 - Benchmark Source		 134.64 μs 163.41 μs 1.21 0.03 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 144.34 μs 190.21 μs 1.32 0.01 False
GetString - Duration of single invocation
📝 - Benchmark Source		 319.91 μs 361.24 μs 1.13 0.00 False
GetString - Duration of single invocation
📝 - Benchmark Source		 200.01 μs 236.03 μs 1.18 0.00 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 159.97 μs 208.18 μs 1.30 0.02 False
GetString - Duration of single invocation
📝 - Benchmark Source		 187.68 μs 251.38 μs 1.34 0.05 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 203.39 μs 238.89 μs 1.17 0.02 False
GetString - Duration of single invocation
📝 - Benchmark Source		 44.30 μs 135.22 μs 3.05 0.01 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 42.38 μs 78.88 μs 1.86 0.04 False
GetString - Duration of single invocation
📝 - Benchmark Source		 261.72 μs 306.76 μs 1.17 0.01 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 163.41458333333333 > 141.77467636148876.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 4/4/2023 10:11:10 PM, 5/19/2023 4:22:12 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -156.9943601397984 (T) = (0 -163313.67105263157) / Math.Sqrt((332399.08210140927 / (15)) + (20366.576792853983 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.20901931770273424 = (135079.45544074927 - 163313.67105263157) / 135079.45544074927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.21237931910568 > 151.46738830222137.
IsChangePoint: Marked as a change because one of 5/6/2023 11:12:53 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -57.93111328415174 (T) = (0 -191002.3032508227) / Math.Sqrt((264013.2822441723 / (15)) + (1247959.6357985316 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.32089916063131374 = (144600.21547711076 - 191002.3032508227) / 144600.21547711076 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Greek)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 361.24470930232553 > 335.2484196428572.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 4/18/2023 4:36:42 PM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -14.118172520751896 (T) = (0 -364439.96422182466) / Math.Sqrt((156432.7000792867 / (15)) + (20419308.001166407 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.14135101329335703 = (319305.770071765 - 364439.96422182466) / 319305.770071765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 236.02786796536802 > 209.01690668706294.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -31.45956270973919 (T) = (0 -237201.51176948054) / Math.Sqrt((948388.27929041 / (15)) + (2754879.557863424 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.18932988111238652 = (199441.31189878515 - 237201.51176948054) / 199441.31189878515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.17916666666665 > 167.74164674908423.
IsChangePoint: Marked as a change because one of 5/6/2023 11:12:53 AM, 5/19/2023 4:22:12 AM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -105.10239953388513 (T) = (0 -207929.19060106558) / Math.Sqrt((2181557.5854358426 / (15)) + (124976.06674677305 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.2995270615430891 = (160003.7404024242 - 207929.19060106558) / 160003.7404024242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishMostlyAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 251.37670546558707 > 199.26802149662032.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -49.931345545245065 (T) = (0 -252501.08962795482) / Math.Sqrt((3578319.5423276178 / (15)) + (2528479.4891668903 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.31998419269850836 = (191290.99501695886 - 252501.08962795482) / 191290.99501695886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: Greek)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 238.88602810650892 > 213.3333234577922.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 3/14/2023 8:42:09 PM, 3/21/2023 2:53:21 PM, 4/4/2023 10:11:10 PM, 5/19/2023 4:22:12 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -46.44101491427244 (T) = (0 -238147.19893087685) / Math.Sqrt((337750.0174463117 / (15)) + (1091737.1015303673 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.17236098584785833 = (203134.70151742332 - 238147.19893087685) / 203134.70151742332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.2222883064516 > 46.67656657750623.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -10.519272418769509 (T) = (0 -144750.35992703532) / Math.Sqrt((105388.31582920565 / (15)) + (181568297.61394644 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.2514945147506236 = (44518.10060584927 - 144750.35992703532) / 44518.10060584927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishAllAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 78.88215829439251 > 44.312702125.
IsChangePoint: Marked as a change because one of 4/11/2023 5:52:15 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -20.02921160723858 (T) = (0 -77139.65478125423) / Math.Sqrt((224515.86856747864 / (15)) + (6072636.986598566 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.8300022207196299 = (42152.765667639374 - 77139.65478125423) / 42152.765667639374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.76269403594773 > 275.3442643065268.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 4/25/2023 6:46:56 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -108.94988157600964 (T) = (0 -306379.07110760966) / Math.Sqrt((241114.1247489303 / (15)) + (294333.10229330295 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.16778130939409266 = (262359.9715485903 - 306379.07110760966) / 262359.9715485903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
ToString_MultipleSegments - Duration of single invocation
📝 - Benchmark Source		 21.13 μs 96.64 μs 4.57 0.07 False
ToString_SingleSegment - Duration of single invocation
📝 - Benchmark Source		 18.98 μs 79.88 μs 4.21 0.02 False
ctor_capacity - Duration of single invocation
📝 - Benchmark Source		 5.52 μs 10.16 μs 1.84 0.05 False
Append_Char_Capacity - Duration of single invocation
📝 - Benchmark Source		 284.55 μs 420.04 μs 1.48 0.48 False
ctor_string - Duration of single invocation
📝 - Benchmark Source		 14.37 μs 72.14 μs 5.02 0.01 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 96.64492912371134 > 22.19193447327905.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -8.8614939204825 (T) = (0 -89009.10485155522) / Math.Sqrt((301034.7288344912 / (15)) + (116611624.63049658 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.172435378856418 = (21332.650303610182 - 89009.10485155522) / 21332.650303610182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ToString_SingleSegment(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.88275290697675 > 20.413894121914147.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -29.62250119071027 (T) = (0 -77905.38003955394) / Math.Sqrt((50497.475381388256 / (15)) + (7820006.9136397075 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.03541094841131 = (19305.439033470506 - 77905.38003955394) / 19305.439033470506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.163017519393357 > 5.795809405674804.
IsChangePoint: Marked as a change because one of 4/25/2023 1:02:54 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -135.76333695931683 (T) = (0 -10131.36906001631) / Math.Sqrt((2180.440353936697 / (15)) + (2003.2499618812392 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.8309336596506445 = (5533.444101928548 - 10131.36906001631) / 5533.444101928548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 420.04497722672073 > 300.31783359375.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 3/30/2023 5:15:39 PM, 5/18/2023 7:10:13 PM, 6/1/2023 5:52:46 AM, 6/28/2023 11:43:34 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -18.9499134774535 (T) = (0 -420434.1379859472) / Math.Sqrt((781893432.2642138 / (15)) + (302892.19304345635 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.48343140528585415 = (283420.00613431155 - 420434.1379859472) / 283420.00613431155 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ctor_string(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.13512505671508 > 15.111548469023417.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -71.20252621282843 (T) = (0 -72955.43077953125) / Math.Sqrt((41195.66829324731 / (15)) + (1345802.9577699201 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -4.056335134774942 = (14428.51963624411 - 72955.43077953125) / 14428.51963624411 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Diagnostics.Perf_Process

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
GetProcesses - Duration of single invocation
📝 - Benchmark Source		 1.07 ms 2.02 ms 1.89 0.64 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Diagnostics.Perf_Process*'

Payloads

Baseline
Compare

System.Diagnostics.Perf_Process.GetProcesses

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0161652529761906 > 1.1468048437500005.
IsChangePoint: Marked as a change because one of 4/8/2023 9:16:19 PM, 4/27/2023 1:05:59 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -11.027556022158558 (T) = (0 -1960596.8091804031) / Math.Sqrt((30159161263.010315 / (15)) + (6175703891.771193 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.6711666399223366 = (1173190.4900109286 - 1960596.8091804031) / 1173190.4900109286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
AllocateUninitializedArray - Duration of single invocation 384.51 ns 1.72 μs 4.47 0.09 False
AllocateArray - Duration of single invocation 775.27 ns 1.88 μs 2.42 0.10 False
AllocateUninitializedArray - Duration of single invocation 155.52 ns 472.17 ns 3.04 0.11 False
AllocateArray - Duration of single invocation 178.93 ns 470.77 ns 2.63 0.15 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7177633974690083 > 399.6578134529415.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -256.67053966083284 (T) = (0 -1722.6061554861167) / Math.Sqrt((56.16090401014884 / (15)) + (46.90461042453306 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.485184576382786 = (384.0658341145383 - 1722.6061554861167) / 384.0658341145383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.877226064390347 > 814.3475494617968.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -440.67129718721765 (T) = (0 -1874.8971003547222) / Math.Sqrt((11.847738386678492 / (15)) + (10.848146958465723 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.414850080560727 = (776.4031048748882 - 1874.8971003547222) / 776.4031048748882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 472.16513668101777 > 162.20111444090492.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 6/8/2023 5:31:08 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -21.318217154468464 (T) = (0 -457.9533767902344) / Math.Sqrt((1.2117547655583252 / (15)) + (403.94823838655543 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.9559985600646497 = (154.92340996952944 - 457.9533767902344) / 154.92340996952944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 470.7704826208066 > 187.7223194932085.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 6/8/2023 5:31:08 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -115.38793741220125 (T) = (0 -468.94846461482723) / Math.Sqrt((43.88253710633865 / (15)) + (6.639499228225897 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.5967833600469057 = (180.5882122590144 - 468.94846461482723) / 180.5882122590144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
AllocateArray - Duration of single invocation 1.47 μs 2.28 μs 1.55 0.07 False
AllocateArray - Duration of single invocation 235.63 ns 797.53 ns 3.38 0.11 False
AllocateUninitializedArray - Duration of single invocation 186.48 ns 788.87 ns 4.23 0.09 False
AllocateUninitializedArray - Duration of single invocation 621.92 ns 2.38 μs 3.83 0.03 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Tests.Perf_GC<Char>.AllocateArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2832381637321504 > 1.5367268810257242.
IsChangePoint: Marked as a change because one of 4/26/2023 8:38:54 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -55.78707820712384 (T) = (0 -2298.0090499132884) / Math.Sqrt((63.27093197615269 / (15)) + (436.35815715226346 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.56747575912705 = (1466.0571536960056 - 2298.0090499132884) / 1466.0571536960056 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 797.532396788991 > 250.09650954960767.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -164.6218631515267 (T) = (0 -800.9410409499897) / Math.Sqrt((1.3949408754027728 / (15)) + (23.237710032619965 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.371495773855683 = (237.56252259335443 - 800.9410409499897) / 237.56252259335443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 788.8731905186607 > 194.27387716710592.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -53.6701390096785 (T) = (0 -777.8408674296243) / Math.Sqrt((1.1605997238051653 / (15)) + (243.4243054817749 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.192164061129945 = (185.54638036278737 - 777.8408674296243) / 185.54638036278737 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3790466059342608 > 660.7370015602506.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -29.929305140791712 (T) = (0 -2322.4811732723774) / Math.Sqrt((44.988564752216064 / (15)) + (6399.296344452154 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.6939609501352844 = (628.7238020714109 - 2322.4811732723774) / 628.7238020714109 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-arm64 os-windows PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jul 20, 2023
@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
ReadAllBytesAsync - Duration of single invocation
📝 - Benchmark Source		 274.91 μs 577.99 μs 2.10 0.27 False
ReadAllBytes - Duration of single invocation
📝 - Benchmark Source		 202.83 μs 479.94 μs 2.37 0.29 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'

Payloads

Baseline
Compare

System.IO.Tests.Perf_File.ReadAllBytesAsync(size: 1048576)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 577.9861458333332 > 290.3141094388058.
IsChangePoint: Marked as a change because one of 4/25/2023 1:02:54 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -15.284701840479862 (T) = (0 -560749.06445802) / Math.Sqrt((487603873.49851245 / (15)) + (594233948.6783401 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.9797180198262446 = (283246.93660526245 - 560749.06445802) / 283246.93660526245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.Perf_File.ReadAllBytes(size: 1048576)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 479.94292338709687 > 231.82580033365883.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -23.577822242410264 (T) = (0 -490412.2564728353) / Math.Sqrt((491765563.06122065 / (15)) + (219213870.52027768 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.3457605174110605 = (209063.22398762486 - 490412.2564728353) / 209063.22398762486 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
CreateFromYawPitchRollBenchmarkBenchmark - Duration of single invocation
📝 - Benchmark Source		 72.09 ns 84.47 ns 1.17 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

System.Numerics.Tests.Perf_Matrix4x4.CreateFromYawPitchRollBenchmarkBenchmark

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 84.46788241404712 > 75.75313546496682.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -7.645247215071372 (T) = (0 -86.27122724084657) / Math.Sqrt((0.7910449094041205 / (15)) + (6.504105128688698 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.19203818135424594 = (72.37287243839447 - 86.27122724084657) / 72.37287243839447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
SystemTextJson_SourceGen_ - Duration of single invocation 139.24 μs 175.28 μs 1.26 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>.SystemTextJson_SourceGen_

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 175.27994382022473 > 145.7066131121954.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -9.778227603968837 (T) = (0 -171882.92275646652) / Math.Sqrt((185977.0733591263 / (15)) + (23079504.215233866 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.2397050631655811 = (138648.23808782734 - 171882.92275646652) / 138648.23808782734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@cincuranet cincuranet removed the untriaged New issue has not been triaged by the area owner label Jul 20, 2023
@cincuranet cincuranet transferred this issue from dotnet/perf-autofiling-issues Jul 20, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jul 20, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jul 20, 2023
@cincuranet
Copy link
Contributor

Best guess for the commit range c4ff86d...4f933e1 is #87374.

cc @MichalPetryka

@AndyAyersMS
Copy link
Member

This is also affecting x64:

image

It's possible that the intrinsifcation here broke static PGO ingestion, but that's just a guess.

@AndyAyersMS
Copy link
Member

It's possible that the intrinsifcation here broke static PGO ingestion, but that's just a guess.

No, that doesn't seem to be the case. Main method in the benchmark above is TranscodeToUtf8 and

base

82.00%   4.265E+07   Tier-1   [System.Private.CoreLib]Utf8Utility.TranscodeToUtf8(wchar*,int32,unsigned int8*,int32,wchar*&,unsigned int8*&)
08.54%   4.44E+06    Tier-1   [System.Private.CoreLib]Utf16Utility.GetPointerToFirstInvalidChar(wchar*,int32,int64&,int32&)
07.25%   3.77E+06    native   coreclr.dll

This method does not end up depending on GetArrayDataReference directly.

Another possibility here is the BDN version update we took with dotnet/performance#3120.

@AndyAyersMS
Copy link
Member

AndyAyersMS commented Jul 20, 2023 •
edited
Loading

Method BDN Toolchain Input Mean Error StdDev Median Min Max Ratio Gen0 Gen1 Gen2 Allocated Alloc Ratio
GetBytes Old BDN \base-rel\corerun.exe Cyrillic 123.3 us 1.10 us 0.97 us 123.3 us 121.9 us 125.0 us 1.00 31.9882 31.9882 31.9882 99.89 KB 1.00
GetBytes Old BDN \diff-rel\corerun.exe Cyrillic 109.4 us 1.35 us 1.26 us 109.4 us 106.0 us 111.3 us 0.89 31.8966 31.8966 31.8966 99.89 KB 1.00
GetBytes New BDN \base-rel\corerun.exe Cyrillic 155.06 us 2.152 us 2.013 us 155.07 us 152.11 us 158.99 us 1.00 0.00 32.1602 32.1602 32.1602 99.89 KB
GetBytes New BDN \diff-rel\corerun.exe Cyrillic 154.99 us 1.623 us 1.519 us 155.06 us 152.45 us 157.82 us 1.00 0.02 32.1782 32.1782 32.1782 99.89 KB

Here

Old BDN is BenchmarkDotNet v0.13.2.2052-nightly
New BDN is BenchmarkDotNet v0.13.7-nightly.20230717.35

So that seems to be the most likely explanation for the "regression"

cc @cincuranet

@AndyAyersMS
Copy link
Member

Also note the operation count dropped quite a bit. Note this can be an effect (not a cause) if the benchmark is indeed running more slowly, but it can also impact the timing.

newplot (70)

@AndyAyersMS AndyAyersMS added tenet-performance-benchmarks Issue from performance benchmark area-Infrastructure and removed untriaged New issue has not been triaged by the area owner needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jul 20, 2023
@ghost
Copy link

ghost commented Jul 20, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
GetBytes - Duration of single invocation
📝 - Benchmark Source		 134.64 μs 163.41 μs 1.21 0.03 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 144.34 μs 190.21 μs 1.32 0.01 False
GetString - Duration of single invocation
📝 - Benchmark Source		 319.91 μs 361.24 μs 1.13 0.00 False
GetString - Duration of single invocation
📝 - Benchmark Source		 200.01 μs 236.03 μs 1.18 0.00 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 159.97 μs 208.18 μs 1.30 0.02 False
GetString - Duration of single invocation
📝 - Benchmark Source		 187.68 μs 251.38 μs 1.34 0.05 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 203.39 μs 238.89 μs 1.17 0.02 False
GetString - Duration of single invocation
📝 - Benchmark Source		 44.30 μs 135.22 μs 3.05 0.01 False
GetBytes - Duration of single invocation
📝 - Benchmark Source		 42.38 μs 78.88 μs 1.86 0.04 False
GetString - Duration of single invocation
📝 - Benchmark Source		 261.72 μs 306.76 μs 1.17 0.01 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

System.Text.Perf_Utf8Encoding.GetBytes(Input: Cyrillic)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 163.41458333333333 > 141.77467636148876.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 4/4/2023 10:11:10 PM, 5/19/2023 4:22:12 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -156.9943601397984 (T) = (0 -163313.67105263157) / Math.Sqrt((332399.08210140927 / (15)) + (20366.576792853983 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.20901931770273424 = (135079.45544074927 - 163313.67105263157) / 135079.45544074927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishMostlyAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.21237931910568 > 151.46738830222137.
IsChangePoint: Marked as a change because one of 5/6/2023 11:12:53 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -57.93111328415174 (T) = (0 -191002.3032508227) / Math.Sqrt((264013.2822441723 / (15)) + (1247959.6357985316 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.32089916063131374 = (144600.21547711076 - 191002.3032508227) / 144600.21547711076 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Greek)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 361.24470930232553 > 335.2484196428572.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 4/18/2023 4:36:42 PM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -14.118172520751896 (T) = (0 -364439.96422182466) / Math.Sqrt((156432.7000792867 / (15)) + (20419308.001166407 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.14135101329335703 = (319305.770071765 - 364439.96422182466) / 319305.770071765 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 236.02786796536802 > 209.01690668706294.
IsChangePoint: Marked as a change because one of 5/19/2023 4:22:12 AM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -31.45956270973919 (T) = (0 -237201.51176948054) / Math.Sqrt((948388.27929041 / (15)) + (2754879.557863424 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.18932988111238652 = (199441.31189878515 - 237201.51176948054) / 199441.31189878515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.17916666666665 > 167.74164674908423.
IsChangePoint: Marked as a change because one of 5/6/2023 11:12:53 AM, 5/19/2023 4:22:12 AM, 6/22/2023 12:06:43 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -105.10239953388513 (T) = (0 -207929.19060106558) / Math.Sqrt((2181557.5854358426 / (15)) + (124976.06674677305 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.2995270615430891 = (160003.7404024242 - 207929.19060106558) / 160003.7404024242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishMostlyAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 251.37670546558707 > 199.26802149662032.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -49.931345545245065 (T) = (0 -252501.08962795482) / Math.Sqrt((3578319.5423276178 / (15)) + (2528479.4891668903 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.31998419269850836 = (191290.99501695886 - 252501.08962795482) / 191290.99501695886 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: Greek)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 238.88602810650892 > 213.3333234577922.
IsChangePoint: Marked as a change because one of 3/6/2023 6:03:21 AM, 3/14/2023 8:42:09 PM, 3/21/2023 2:53:21 PM, 4/4/2023 10:11:10 PM, 5/19/2023 4:22:12 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -46.44101491427244 (T) = (0 -238147.19893087685) / Math.Sqrt((337750.0174463117 / (15)) + (1091737.1015303673 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.17236098584785833 = (203134.70151742332 - 238147.19893087685) / 203134.70151742332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.2222883064516 > 46.67656657750623.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -10.519272418769509 (T) = (0 -144750.35992703532) / Math.Sqrt((105388.31582920565 / (15)) + (181568297.61394644 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.2514945147506236 = (44518.10060584927 - 144750.35992703532) / 44518.10060584927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetBytes(Input: EnglishAllAscii)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 78.88215829439251 > 44.312702125.
IsChangePoint: Marked as a change because one of 4/11/2023 5:52:15 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -20.02921160723858 (T) = (0 -77139.65478125423) / Math.Sqrt((224515.86856747864 / (15)) + (6072636.986598566 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.8300022207196299 = (42152.765667639374 - 77139.65478125423) / 42152.765667639374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.76269403594773 > 275.3442643065268.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 4/25/2023 6:46:56 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -108.94988157600964 (T) = (0 -306379.07110760966) / Math.Sqrt((241114.1247489303 / (15)) + (294333.10229330295 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.16778130939409266 = (262359.9715485903 - 306379.07110760966) / 262359.9715485903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
ToString_MultipleSegments - Duration of single invocation
📝 - Benchmark Source		 21.13 μs 96.64 μs 4.57 0.07 False
ToString_SingleSegment - Duration of single invocation
📝 - Benchmark Source		 18.98 μs 79.88 μs 4.21 0.02 False
ctor_capacity - Duration of single invocation
📝 - Benchmark Source		 5.52 μs 10.16 μs 1.84 0.05 False
Append_Char_Capacity - Duration of single invocation
📝 - Benchmark Source		 284.55 μs 420.04 μs 1.48 0.48 False
ctor_string - Duration of single invocation
📝 - Benchmark Source		 14.37 μs 72.14 μs 5.02 0.01 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 96.64492912371134 > 22.19193447327905.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -8.8614939204825 (T) = (0 -89009.10485155522) / Math.Sqrt((301034.7288344912 / (15)) + (116611624.63049658 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.172435378856418 = (21332.650303610182 - 89009.10485155522) / 21332.650303610182 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ToString_SingleSegment(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.88275290697675 > 20.413894121914147.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -29.62250119071027 (T) = (0 -77905.38003955394) / Math.Sqrt((50497.475381388256 / (15)) + (7820006.9136397075 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.03541094841131 = (19305.439033470506 - 77905.38003955394) / 19305.439033470506 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ctor_capacity(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.163017519393357 > 5.795809405674804.
IsChangePoint: Marked as a change because one of 4/25/2023 1:02:54 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -135.76333695931683 (T) = (0 -10131.36906001631) / Math.Sqrt((2180.440353936697 / (15)) + (2003.2499618812392 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.8309336596506445 = (5533.444101928548 - 10131.36906001631) / 5533.444101928548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 420.04497722672073 > 300.31783359375.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 3/30/2023 5:15:39 PM, 5/18/2023 7:10:13 PM, 6/1/2023 5:52:46 AM, 6/28/2023 11:43:34 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -18.9499134774535 (T) = (0 -420434.1379859472) / Math.Sqrt((781893432.2642138 / (15)) + (302892.19304345635 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.48343140528585415 = (283420.00613431155 - 420434.1379859472) / 283420.00613431155 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Tests.Perf_StringBuilder.ctor_string(length: 100000)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.13512505671508 > 15.111548469023417.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -71.20252621282843 (T) = (0 -72955.43077953125) / Math.Sqrt((41195.66829324731 / (15)) + (1345802.9577699201 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -4.056335134774942 = (14428.51963624411 - 72955.43077953125) / 14428.51963624411 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Diagnostics.Perf_Process

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
GetProcesses - Duration of single invocation
📝 - Benchmark Source		 1.07 ms 2.02 ms 1.89 0.64 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Diagnostics.Perf_Process*'

Payloads

Baseline
Compare

System.Diagnostics.Perf_Process.GetProcesses

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.0161652529761906 > 1.1468048437500005.
IsChangePoint: Marked as a change because one of 4/8/2023 9:16:19 PM, 4/27/2023 1:05:59 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -11.027556022158558 (T) = (0 -1960596.8091804031) / Math.Sqrt((30159161263.010315 / (15)) + (6175703891.771193 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.6711666399223366 = (1173190.4900109286 - 1960596.8091804031) / 1173190.4900109286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
AllocateUninitializedArray - Duration of single invocation 384.51 ns 1.72 μs 4.47 0.09 False
AllocateArray - Duration of single invocation 775.27 ns 1.88 μs 2.42 0.10 False
AllocateUninitializedArray - Duration of single invocation 155.52 ns 472.17 ns 3.04 0.11 False
AllocateArray - Duration of single invocation 178.93 ns 470.77 ns 2.63 0.15 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7177633974690083 > 399.6578134529415.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -256.67053966083284 (T) = (0 -1722.6061554861167) / Math.Sqrt((56.16090401014884 / (15)) + (46.90461042453306 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.485184576382786 = (384.0658341145383 - 1722.6061554861167) / 384.0658341145383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.877226064390347 > 814.3475494617968.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -440.67129718721765 (T) = (0 -1874.8971003547222) / Math.Sqrt((11.847738386678492 / (15)) + (10.848146958465723 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.414850080560727 = (776.4031048748882 - 1874.8971003547222) / 776.4031048748882 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateUninitializedArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 472.16513668101777 > 162.20111444090492.
IsChangePoint: Marked as a change because one of 5/23/2023 1:41:12 PM, 6/8/2023 5:31:08 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -21.318217154468464 (T) = (0 -457.9533767902344) / Math.Sqrt((1.2117547655583252 / (15)) + (403.94823838655543 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.9559985600646497 = (154.92340996952944 - 457.9533767902344) / 154.92340996952944 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Byte>.AllocateArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 470.7704826208066 > 187.7223194932085.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/23/2023 1:41:12 PM, 6/8/2023 5:31:08 AM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -115.38793741220125 (T) = (0 -468.94846461482723) / Math.Sqrt((43.88253710633865 / (15)) + (6.639499228225897 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -1.5967833600469057 = (180.5882122590144 - 468.94846461482723) / 180.5882122590144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture arm64
OS Windows 10.0.19041
Queue SurfaceWindows
Baseline bb38848e9877dc0b43860dc219ebe36fc04a6bb3
Compare 16dc5330051a97aec4ba89d21263cdccc7b7552c
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
AllocateArray - Duration of single invocation 1.47 μs 2.28 μs 1.55 0.07 False
AllocateArray - Duration of single invocation 235.63 ns 797.53 ns 3.38 0.11 False
AllocateUninitializedArray - Duration of single invocation 186.48 ns 788.87 ns 4.23 0.09 False
AllocateUninitializedArray - Duration of single invocation 621.92 ns 2.38 μs 3.83 0.03 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

System.Tests.Perf_GC<Char>.AllocateArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.2832381637321504 > 1.5367268810257242.
IsChangePoint: Marked as a change because one of 4/26/2023 8:38:54 PM, 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -55.78707820712384 (T) = (0 -2298.0090499132884) / Math.Sqrt((63.27093197615269 / (15)) + (436.35815715226346 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -0.56747575912705 = (1466.0571536960056 - 2298.0090499132884) / 1466.0571536960056 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 797.532396788991 > 250.09650954960767.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -164.6218631515267 (T) = (0 -800.9410409499897) / Math.Sqrt((1.3949408754027728 / (15)) + (23.237710032619965 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.371495773855683 = (237.56252259335443 - 800.9410409499897) / 237.56252259335443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 1000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 788.8731905186607 > 194.27387716710592.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -53.6701390096785 (T) = (0 -777.8408674296243) / Math.Sqrt((1.1605997238051653 / (15)) + (243.4243054817749 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -3.192164061129945 = (185.54638036278737 - 777.8408674296243) / 185.54638036278737 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: True)

ETL Files

Histogram

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3790466059342608 > 660.7370015602506.
IsChangePoint: Marked as a change because one of 7/17/2023 4:07:27 PM, 7/19/2023 6:59:29 PM falls between 7/10/2023 2:17:11 PM and 7/19/2023 6:59:29 PM.
IsRegressionStdDev: Marked as regression because -29.929305140791712 (T) = (0 -2322.4811732723774) / Math.Sqrt((44.988564752216064 / (15)) + (6399.296344452154 / (2))) is less than -2.131449545559758 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (15) + (2) - 2, .025) and -2.6939609501352844 = (628.7238020714109 - 2322.4811732723774) / 628.7238020714109 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

arch-arm64, os-windows, tenet-performance-benchmarks, area-Infrastructure, runtime-coreclr, PGO, needs-area-label
Milestone: -

@AndyAyersMS AndyAyersMS mentioned this issue Jul 23, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@agocke agocke closed this as completed Jun 18, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Jul 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 area-Infrastructure os-windows PGO runtime-coreclr specific to the CoreCLR runtime tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@agocke @cincuranet @AndyAyersMS