Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in multiple benchmarks after char.Is* update #69727

Closed
performanceautofiler bot opened this issue May 24, 2022 · 20 comments
Closed

Regressions in multiple benchmarks after char.Is* update #69727

performanceautofiler bot opened this issue May 24, 2022 · 20 comments
Assignees
@tannergooding
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.72 ns 72.62 ns 1.07 0.02 False
MultiplyOperatorBenchmark - Duration of single invocation 51.64 ns 64.82 ns 1.26 0.03 False
DivisionOperatorBenchmark - Duration of single invocation 110.14 ns 115.99 ns 1.05 0.01 False
GetHashCodeBenchmark - Duration of single invocation 102.10 ns 114.58 ns 1.12 0.04 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.SquareRootBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.616263426816 > 70.3165027262756.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -45.0318918142146 (T) = (0 -72.8527869121108) / Math.Sqrt((0.10060373577538348 / (33)) + (0.29639641692658697 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.08162207084295842 = (67.35512234447401 - 72.8527869121108) / 67.35512234447401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.81981764655713 > 53.46745717697251.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -28.529296478118287 (T) = (0 -65.17046109518371) / Math.Sqrt((7.8020485166881395 / (43)) + (0.030602190839587127 / (14))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (14) - 2, .025) and -0.23090626675282228 = (52.94510463993808 - 65.17046109518371) / 52.94510463993808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.DivisionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.98556839828156 > 115.70109878652117.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -155.80186125871447 (T) = (0 -116.382354674369) / Math.Sqrt((0.007821590082532347 / (31)) + (0.03303517114606407 / (25))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (25) - 2, .025) and -0.05608511730511969 = (110.2016804965014 - 116.382354674369) / 110.2016804965014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.57668798889473 > 107.65639252344216.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -283.03839865316087 (T) = (0 -114.59128121623345) / Math.Sqrt((0.07882115090493173 / (44)) + (0.0012095622524388827 / (14))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (14) - 2, .025) and -0.11986198350643835 = (102.32625350619793 - 114.59128121623345) / 102.32625350619793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.56 ns 72.62 ns 1.07 0.01 False
GetHashCodeBenchmark - Duration of single invocation 105.18 ns 114.93 ns 1.09 0.06 False
MultiplyOperatorBenchmark - Duration of single invocation 50.92 ns 65.31 ns 1.28 0.07 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.SquareRootBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.61622324189207 > 70.29034196284785.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.44383568394873 (T) = (0 -73.07887024394701) / Math.Sqrt((0.4275740748536778 / (33)) + (0.10853637117538197 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08529938753935677 = (67.33521743676181 - 73.07887024394701) / 67.33521743676181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.93390136649963 > 107.37540770947281.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -48.89414960870857 (T) = (0 -114.77622672101295) / Math.Sqrt((2.158761094640268 / (44)) + (0.028601638891270608 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.10674676117540412 = (103.70595220817863 - 114.77622672101295) / 103.70595220817863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.30981681633082 > 53.33145200677543.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -15.112172425806873 (T) = (0 -62.63261316978918) / Math.Sqrt((0.11165366968483725 / (32)) + (14.904766609836882 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.22978084202205626 = (50.92989826285313 - 62.63261316978918) / 50.92989826285313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 244.70 ns 295.96 ns 1.21 0.06 False
Concat_CharEnumerable - Duration of single invocation 7.03 μs 8.46 μs 1.20 0.06 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 295.96439696558804 > 258.5723078295418.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.214416060613296 (T) = (0 -297.6240622174258) / Math.Sqrt((14.284845301254313 / (33)) + (23.17517385721064 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.2073050297315947 = (246.5193591412378 - 297.6240622174258) / 246.5193591412378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Concat_CharEnumerable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.459203054806828 > 7.38004250196541.
IsChangePoint: Marked as a change because one of 3/15/2022 11:00:34 PM, 3/23/2022 6:35:24 PM, 3/29/2022 11:07:04 AM, 5/5/2022 12:13:02 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -190.01315732087696 (T) = (0 -8472.773343347046) / Math.Sqrt((78.21027311391161 / (44)) + (730.5417624546898 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.20591487712144863 = (7026.0127842288375 - 8472.773343347046) / 7026.0127842288375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 0c6d412f86205043726e273127d289b65d5b6c7a
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 975.11 ns 1.61 μs 1.65 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6128745088408643 > 1023.9368522813845.
IsChangePoint: Marked as a change because one of 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -14.572122467414548 (T) = (0 -1455.1143152156847) / Math.Sqrt((0.005368131784794816 / (33)) + (26039.113951732077 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.4922315216536375 = (975.1263755661582 - 1455.1143152156847) / 975.1263755661582 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels May 24, 2022
@kunalspathak kunalspathak changed the title [Perf] Changes at 5/19/2022 4:33:45 PM Regressions in multiple benchmarks after roslyn update May 24, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues May 24, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 24, 2022
@kunalspathak kunalspathak removed their assignment May 24, 2022
@ghost
Copy link

ghost commented May 24, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.72 ns 72.62 ns 1.07 0.02 False
MultiplyOperatorBenchmark - Duration of single invocation 51.64 ns 64.82 ns 1.26 0.03 False
DivisionOperatorBenchmark - Duration of single invocation 110.14 ns 115.99 ns 1.05 0.01 False
GetHashCodeBenchmark - Duration of single invocation 102.10 ns 114.58 ns 1.12 0.04 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.SquareRootBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.616263426816 > 70.3165027262756.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -45.0318918142146 (T) = (0 -72.8527869121108) / Math.Sqrt((0.10060373577538348 / (33)) + (0.29639641692658697 / (25))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (25) - 2, .025) and -0.08162207084295842 = (67.35512234447401 - 72.8527869121108) / 67.35512234447401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.81981764655713 > 53.46745717697251.
IsChangePoint: Marked as a change because one of 3/11/2022 8:42:43 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -28.529296478118287 (T) = (0 -65.17046109518371) / Math.Sqrt((7.8020485166881395 / (43)) + (0.030602190839587127 / (14))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (43) + (14) - 2, .025) and -0.23090626675282228 = (52.94510463993808 - 65.17046109518371) / 52.94510463993808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.DivisionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 115.98556839828156 > 115.70109878652117.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -155.80186125871447 (T) = (0 -116.382354674369) / Math.Sqrt((0.007821590082532347 / (31)) + (0.03303517114606407 / (25))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (25) - 2, .025) and -0.05608511730511969 = (110.2016804965014 - 116.382354674369) / 110.2016804965014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.57668798889473 > 107.65639252344216.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -283.03839865316087 (T) = (0 -114.59128121623345) / Math.Sqrt((0.07882115090493173 / (44)) + (0.0012095622524388827 / (14))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (14) - 2, .025) and -0.11986198350643835 = (102.32625350619793 - 114.59128121623345) / 102.32625350619793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SquareRootBenchmark - Duration of single invocation 67.56 ns 72.62 ns 1.07 0.01 False
GetHashCodeBenchmark - Duration of single invocation 105.18 ns 114.93 ns 1.09 0.06 False
MultiplyOperatorBenchmark - Duration of single invocation 50.92 ns 65.31 ns 1.28 0.07 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.SquareRootBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 72.61622324189207 > 70.29034196284785.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.44383568394873 (T) = (0 -73.07887024394701) / Math.Sqrt((0.4275740748536778 / (33)) + (0.10853637117538197 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.08529938753935677 = (67.33521743676181 - 73.07887024394701) / 67.33521743676181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.GetHashCodeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.93390136649963 > 107.37540770947281.
IsChangePoint: Marked as a change because one of 4/25/2022 11:05:49 AM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -48.89414960870857 (T) = (0 -114.77622672101295) / Math.Sqrt((2.158761094640268 / (44)) + (0.028601638891270608 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.10674676117540412 = (103.70595220817863 - 114.77622672101295) / 103.70595220817863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.MultiplyOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 65.30981681633082 > 53.33145200677543.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 3/29/2022 11:07:04 AM, 4/25/2022 11:05:49 AM, 5/11/2022 7:02:26 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -15.112172425806873 (T) = (0 -62.63261316978918) / Math.Sqrt((0.11165366968483725 / (32)) + (14.904766609836882 / (25))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (25) - 2, .025) and -0.22978084202205626 = (50.92989826285313 - 62.63261316978918) / 50.92989826285313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 15dda0efe52c2aaa7a4e783f443eefb558529248
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Format_MultipleArgs - Duration of single invocation 244.70 ns 295.96 ns 1.21 0.06 False
Concat_CharEnumerable - Duration of single invocation 7.03 μs 8.46 μs 1.20 0.06 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 295.96439696558804 > 258.5723078295418.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -43.214416060613296 (T) = (0 -297.6240622174258) / Math.Sqrt((14.284845301254313 / (33)) + (23.17517385721064 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.2073050297315947 = (246.5193591412378 - 297.6240622174258) / 246.5193591412378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Concat_CharEnumerable

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.459203054806828 > 7.38004250196541.
IsChangePoint: Marked as a change because one of 3/15/2022 11:00:34 PM, 3/23/2022 6:35:24 PM, 3/29/2022 11:07:04 AM, 5/5/2022 12:13:02 AM, 5/11/2022 7:02:26 PM, 5/20/2022 10:18:01 PM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -190.01315732087696 (T) = (0 -8472.773343347046) / Math.Sqrt((78.21027311391161 / (44)) + (730.5417624546898 / (13))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (44) + (13) - 2, .025) and -0.20591487712144863 = (7026.0127842288375 - 8472.773343347046) / 7026.0127842288375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 6e4eba8c98f4e05ece31a883dc23076a78f7c7b9
Compare 0c6d412f86205043726e273127d289b65d5b6c7a
Diff Diff

Regressions in System.Globalization.Tests.StringEquality

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Compare_Same - Duration of single invocation 975.11 ns 1.61 μs 1.65 0.01 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_Same(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.6128745088408643 > 1023.9368522813845.
IsChangePoint: Marked as a change because one of 5/18/2022 10:04:15 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 6:57:22 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -14.572122467414548 (T) = (0 -1455.1143152156847) / Math.Sqrt((0.005368131784794816 / (33)) + (26039.113951732077 / (24))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (24) - 2, .025) and -0.4922315216536375 = (975.1263755661582 - 1455.1143152156847) / 975.1263755661582 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kunalspathak
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.18362, Regression, CoreClr, x64
Milestone: -

@kunalspathak kunalspathak added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels May 24, 2022
@kunalspathak
Copy link
Member

kunalspathak commented May 24, 2022 •
edited

Several regressions in windows/x64 dotnet/perf-autofiling-issues#5457, dotnet/perf-autofiling-issues#5458, dotnet/perf-autofiling-issues#5471, dotnet/perf-autofiling-issues#5489

@kunalspathak
Copy link
Member

#69627

This was referenced May 24, 2022
Closed
Closed
Closed
@kunalspathak
Copy link
Member

Ubuntu/x64 regressions: dotnet/perf-autofiling-issues#5473

This was referenced May 24, 2022
Closed
Closed
@tannergooding
Copy link
Member

@kunalspathak this diff range doesn't look to include the Roslyn update.

I'm taking a look, but I think these might be unrelated.

@kunalspathak
Copy link
Member

I can see e4ac5cd...7626c5d is the only range in https://pvscmdupload.blob.core.windows.net/autofilereport/autofilereports/05_24_2022/refs/heads/main_x64_Windows%2010.0.18362_Regression/System.Numerics.Tests.Perf_VectorOf(SByte).html

@tannergooding
Copy link
Member

I can see e4ac5cd...7626c5d is the only range in

Ah, I see. You're looking at a different range than what the bot flagged (6e4eba8...15dda0e). Basically looking at a possible secondary increase.

@tannergooding
Copy link
Member

System.Numerics.Tests.Perf_VectorOf has had no changes in the generated assembly.

System.Tests.Perf_String has had changes but they're unrelated to the Roslyn change (no diffs when comparing just that commit to the one prior).

@tannergooding
Copy link
Member

tannergooding commented May 25, 2022 •
edited

The regression is due to #69318. CC. @stephentoub

The disassembly for
String-base.md - 2291ac9
String-diff.md - 9260c24

The first encountered diff looks like the below and is in System.Text.ValueStringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ParamsArray):

       movzx     eax,word ptr [rdi+rax*2+0C]
       cmp       eax,30
       jl        near ptr M03_L59
       cmp       eax,39
       jg        near ptr M03_L59
       xor       edx,edx
M03_L06:
       lea       edx,[rdx+rdx*4]

becomes

       movzx     eax,word ptr [rdi+rax*2+0C]
       movzx     ecx,ax
       add       ecx,0FFFFFFD0
       cmp       ecx,9
       jbe       short M03_L07
M03_L06:
       call      qword ptr [7FFCC6C59438]
       int       3
M03_L07:
       xor       edx,edx
M03_L08:
       lea       edx,[rdx+rdx*4]

The above likely corresponds to: https://github.com/dotnet/runtime/pull/69318/files#diff-072e8df6b3d249d58da2b688cbd9369ae5d1e9fb4c1c1da77a19384f3f7711a0R1528

Rather than the "simple" (ch = format[pos]) < '0' || ch > '9', we now have it call down into https://github.com/dotnet/runtime/pull/69318/files#diff-ea5ef93981b66b93a8906532bba87b7908ebf2042170b647a8953a9e952cbc06R260 and then https://github.com/dotnet/runtime/pull/69318/files#diff-ea5ef93981b66b93a8906532bba87b7908ebf2042170b647a8953a9e952cbc06R322. This is public static bool IsAsciiDigit(char c) => IsBetween(c, '0', '9'); and public static bool IsBetween(char c, char minInclusive, char maxInclusive) =>(uint)(c - minInclusive) <= (uint)(maxInclusive - minInclusive); respectively.

The JIT doesn't seem to handle this as well and places the "error case" as the fallthrough case (meaning most CPUs will predict it take by default). If you follow the rest of the diffs, it looks to likewise be doing less loop unrolling among and placing other error cases "inline" rather than being grouped near the end of the method body as "cold paths".

We could of course fix up the managed side methods, but this likely deserves a closer look by the runtime team to determine why it's getting pessimized like this.

@kunalspathak
Copy link
Member

Thanks @tannergooding for taking a look. The diffs doesn't seem to be working well. Could you please retype them pointing to the source code instead of diffs?

Thing to note, for the change itself to IsBetween(), I can see that we would use 1 less comparison but for cases that are < '0, we will be doing extra work of 2 sub before returning the result.

image

and placing other error cases "inline" rather than being grouped near the end of the method body as "cold paths".

It is tier 0 code. If you see tier 1 code, they are moved towards the end of the method.
formathelper_tier1.md
formathelper_tier0.md

Are all the regressions related to #69318 ?

Several regressions in windows/x64 dotnet/perf-autofiling-issues#5457, dotnet/perf-autofiling-issues#5458, dotnet/perf-autofiling-issues#5471, dotnet/perf-autofiling-issues#5489

Even the ones listed here?

@tannergooding
Copy link
Member

I didn't check every regression in depth, but many of the string related ones were.

It is tier 0 code. If you see tier 1 code, they are moved towards the end of the method.

👍, this must be another case where benchmark.net doesn't grab the right method body for the disassembly.

@stephentoub
Copy link
Member

Thing to note, for the change itself to IsBetween(), I can see that we would use 1 less comparison but for cases that are < '0, we will be doing extra work of 2 sub before returning the result.

Yes, but most of ASCII, and in particular all letters both upper and lower case, are > '0', so the expectation is that the majority of characters evaluated aren't < '0'.

@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label May 25, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 7.0.0 milestone May 25, 2022
@dakersnar
Copy link
Contributor

More regressions, arm64: dotnet/perf-autofiling-issues#5538

@AndyAyersMS
Copy link
Member

AndyAyersMS commented May 27, 2022 •
edited

Some improvements seen too (note autofiler may be filing duplicates...)

@AndyAyersMS
Copy link
Member

Thing to note, for the change itself to IsBetween(), I can see that we would use 1 less comparison but for cases that are < '0, we will be doing extra work of 2 sub before returning the result.

Yes, but most of ASCII, and in particular all letters both upper and lower case, are > '0', so the expectation is that the majority of characters evaluated aren't < '0'.

Could be this gets ironed out with a MPGO update, which should show that test is strongly biased. Our last MPGO update was on 5/9 but the data it gathered was from 4/17.

@EgorBo you mentioned the stale MPGO data the other day -- any idea why we're not getting new data?

@EgorBo
Copy link
Member

EgorBo commented May 27, 2022 •
edited

@EgorBo you mentioned the stale MPGO data the other day -- any idea why we're not getting new data?

I assume the PRs to update dotnet sdk in dotnet-optimization have to be approved (and conflicts need to be resolved) - doing it now

@kunalspathak
Copy link
Member

Regression in ubuntu/arm64 - dotnet/perf-autofiling-issues#5569

@kunalspathak kunalspathak mentioned this issue May 31, 2022
Closed
@dakersnar
Copy link
Contributor

While generating the preview 5 perf report, I'm seeing 0-30% regressions across a variety of configs on the following benchmarks:

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)
System.Text.Perf_Utf8Encoding.GetString(Input: Greek)
System.Text.Perf_Utf8Encoding.GetString(Input: Cyrillic)
System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)

Are we in agreement that this Roslyn update is responsible, or is #69318 actually to blame? Also, let me know if it's worth creating a new issue for these specific regressions.

System.Text.Perf_Utf8Encoding.GetString(Input: Chinese)

Result Ratio Operating System Bit Processor Name
Slower 0.82 debian 11 Arm64 Unknown processor
Slower 0.90 ubuntu 18.04 Arm64 Unknown processor
Slower 0.82 ubuntu 20.04 Arm64 Unknown processor
Slower 0.83 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.87 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 0.90 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.85 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.86 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Slower 0.81 Windows 11 X64 AMD Ryzen 9 5900X
Slower 0.82 Windows 11 X64 AMD Ryzen 9 5950X
Slower 0.89 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.83 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Slower 0.88 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 0.90 ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
Slower 0.76 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.85 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.98 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.88 macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

<\details>

@tannergooding
Copy link
Member

As far as I was able to tell the change has nothing to do with the Roslyn update.

That doesn't necessarily mean everything is the fault of #69318 however.

@dakersnar dakersnar mentioned this issue Jun 10, 2022
Closed
14 tasks
@tannergooding tannergooding changed the title Regressions in multiple benchmarks after roslyn update Regressions in multiple benchmarks after char.Is* update Jun 12, 2022
@tannergooding
Copy link
Member

These look to have been resolved for the most part. They are within the normal/expected noise ranges in most cases.

@dotnet dotnet locked as resolved and limited conversation to collaborators Jul 17, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime arch-x64 and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
8 participants
@EgorBo @jeffhandley @stephentoub @AndyAyersMS @tannergooding @kunalspathak @dakersnar @JulieLeeMSFT