Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 60 Improvements on 2/9/2023 12:40:40 PM #12985

Closed
performanceautofiler bot opened this issue Feb 14, 2023 · 3 comments
Closed

[Perf] Windows/x64: 60 Improvements on 2/9/2023 12:40:40 PM #12985

performanceautofiler bot opened this issue Feb 14, 2023 · 3 comments
Assignees
@EgorBo
Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-improvement runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 51.94 μs 45.68 μs 0.88 0.01 False
ImmutableSortedSet - Duration of single invocation 179.85 μs 164.94 μs 0.92 0.01 False
ImmutableList - Duration of single invocation 101.71 μs 89.08 μs 0.88 0.02 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.SortedSet(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 45.67613022941969 < 50.054110001727324.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 77.79091002341295 (T) = (0 -45967.448897016846) / Math.Sqrt((636384.9539038617 / (299)) + (92727.96894886105 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11877436088007147 = (52163.086111434626 - 45967.448897016846) / 52163.086111434626 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 164.9414298245614 < 168.83490861742422.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 28.2660621643388 (T) = (0 -166082.7369351791) / Math.Sqrt((4395956.838199926 / (299)) + (2980543.748657064 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.0618742612189658 = (177036.75538311191 - 166082.7369351791) / 177036.75538311191 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;Int32&gt;.ImmutableList(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 89.08496428571429 < 96.82860696067956.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 61.842070840397945 (T) = (0 -89169.62147396228) / Math.Sqrt((2307053.8824507645 / (299)) + (872604.1082623651 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.13116046066080025 = (102630.71307940323 - 89169.62147396228) / 102630.71307940323 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsDefinedClassHitInherit - Duration of single invocation 475.86 ns 438.57 ns 0.92 0.07 False Trace Trace
IsDefinedMethodBaseHitInherit - Duration of single invocation 505.33 ns 464.21 ns 0.92 0.06 False
IsDefinedMethodOverrideMissInherit - Duration of single invocation 501.27 ns 469.46 ns 0.94 0.08 False
IsDefinedClassHit - Duration of single invocation 487.33 ns 440.26 ns 0.90 0.12 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Attributes.IsDefinedClassHitInherit

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 438.57259661841033 < 456.7786079876879.
IsChangePoint: Marked as a change because one of 1/9/2023 8:29:54 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 39.06867646681837 (T) = (0 -438.84170280776834) / Math.Sqrt((207.86481978433716 / (299)) + (15.803539921155485 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09571512165522761 = (485.2914311815499 - 438.84170280776834) / 485.2914311815499 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Reflection.Attributes.IsDefinedClassHitInherit()
       mov       rcx,22ECAF79ED8
       mov       rdx,22ECAF79F00
       mov       r8d,1
       jmp       qword ptr [7FFA756193C8]; System.RuntimeType.IsDefined(System.Type, Boolean)
; Total bytes of code 32

; System.RuntimeType.IsDefined(System.Type, Boolean)
       push      rdi
       push      rsi
       sub       rsp,28
       mov       rsi,rcx
       mov       edi,r8d
       test      rdx,rdx
       je        short M01_L02
       mov       rcx,offset MT_System.RuntimeType
       cmp       [rdx],rcx
       jne       short M01_L01
M01_L00:
       test      rdx,rdx
       je        short M01_L03
       mov       r8,offset MT_System.RuntimeType
       cmp       [rdx],r8
       jne       short M01_L03
       movzx     r8d,dil
       mov       rcx,rsi
       add       rsp,28
       pop       rsi
       pop       rdi
       jmp       qword ptr [7FFA757EF000]; System.Reflection.CustomAttribute.IsDefined(System.RuntimeType, System.RuntimeType, Boolean)
M01_L01:
       mov       rcx,rdx
       mov       rax,[rdx]
       mov       rax,[rax+58]
       call      qword ptr [rax]
       mov       rdx,rax
       jmp       short M01_L00
M01_L02:
       mov       ecx,0E9
       mov       rdx,7FFA754F4000
       call      CORINFO_HELP_STRCNS
       mov       rcx,rax
       call      qword ptr [7FFA757E5BA0]
       int       3
M01_L03:
       mov       rcx,offset MT_System.ArgumentException
       call      CORINFO_HELP_NEWSFAST
       mov       rsi,rax
       call      qword ptr [7FFA75785648]
       mov       rdi,rax
       mov       ecx,0E9
       mov       rdx,7FFA754F4000
       call      CORINFO_HELP_STRCNS
       mov       r8,rax
       mov       rdx,rdi
       mov       rcx,rsi
       call      qword ptr [7FFA756BDBD0]
       mov       rcx,rsi
       call      CORINFO_HELP_THROW
       int       3
; Total bytes of code 189

Compare Jit Disasm

; System.Reflection.Attributes.IsDefinedClassHitInherit()
       mov       rcx,21CB106B4F8
       mov       rdx,21CB106B520
       mov       r8d,1
       jmp       qword ptr [7FF8D19B93C8]; System.RuntimeType.IsDefined(System.Type, Boolean)
; Total bytes of code 32
; System.RuntimeType.IsDefined(System.Type, Boolean)
       push      rdi
       push      rsi
       sub       rsp,28
       mov       rsi,rcx
       mov       edi,r8d
       test      rdx,rdx
       je        short M01_L02
       mov       rcx,offset MT_System.RuntimeType
       cmp       [rdx],rcx
       jne       short M01_L01
M01_L00:
       test      rdx,rdx
       je        short M01_L03
       mov       r8,offset MT_System.RuntimeType
       cmp       [rdx],r8
       jne       short M01_L03
       movzx     r8d,dil
       mov       rcx,rsi
       add       rsp,28
       pop       rsi
       pop       rdi
       jmp       qword ptr [7FF8D1B8F000]; System.Reflection.CustomAttribute.IsDefined(System.RuntimeType, System.RuntimeType, Boolean)
M01_L01:
       mov       rcx,rdx
       mov       rax,[rdx]
       mov       rax,[rax+58]
       call      qword ptr [rax]
       mov       rdx,rax
       jmp       short M01_L00
M01_L02:
       mov       ecx,0E9
       mov       rdx,7FF8D1894000
       call      CORINFO_HELP_STRCNS
       mov       rcx,rax
       call      qword ptr [7FF8D1B85BA0]
       int       3
M01_L03:
       mov       rcx,offset MT_System.ArgumentException
       call      CORINFO_HELP_NEWSFAST
       mov       rsi,rax
       call      qword ptr [7FF8D1B25648]
       mov       rdi,rax
       mov       ecx,0E9
       mov       rdx,7FF8D1894000
       call      CORINFO_HELP_STRCNS
       mov       r8,rax
       mov       rdx,rdi
       mov       rcx,rsi
       call      qword ptr [7FF8D1A5DBD0]
       mov       rcx,rsi
       call      CORINFO_HELP_THROW
       int       3
; Total bytes of code 189

System.Reflection.Attributes.IsDefinedMethodBaseHitInherit

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 464.21334104088754 < 479.90053645031685.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 43.203907879421294 (T) = (0 -461.2785938687879) / Math.Sqrt((223.54645906472078 / (299)) + (18.74129844932296 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10590987142709027 = (515.919569098757 - 461.2785938687879) / 515.919569098757 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodOverrideMissInherit

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 469.46392847473214 < 478.48772646293617.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 15.080878984945256 (T) = (0 -469.3222084596509) / Math.Sqrt((213.01727900607807 / (299)) + (182.41604652610516 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08794220685143164 = (514.5750762563807 - 469.3222084596509) / 514.5750762563807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassHit

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 440.26146786817515 < 461.80375843954914.
IsChangePoint: Marked as a change because one of 1/8/2023 4:08:48 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 37.403131333265854 (T) = (0 -439.511091456915) / Math.Sqrt((216.90719751509855 / (299)) + (17.828124148878537 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09540303567479137 = (485.8639911363972 - 439.511091456915) / 485.8639911363972 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array_Comparison - Duration of single invocation 29.66 μs 25.23 μs 0.85 0.01 False
Array_ComparerClass - Duration of single invocation 29.64 μs 25.04 μs 0.84 0.00 False
Array_ComparerStruct - Duration of single invocation 35.00 μs 31.33 μs 0.90 0.01 False
LinqQuery - Duration of single invocation 38.17 μs 33.75 μs 0.88 0.01 False
LinqOrderByExtension - Duration of single invocation 38.33 μs 34.67 μs 0.90 0.01 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<Int32>.Array_Comparison(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 25.229884 < 28.35812238571429.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 173.16249586197134 (T) = (0 -25112.597280885777) / Math.Sqrt((56485.88749501119 / (299)) + (11228.252987832124 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1542223824374112 = (29691.72600388353 - 25112.597280885777) / 29691.72600388353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.Array_ComparerClass(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 25.040995999999996 < 28.205588033333335.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 149.07867568162393 (T) = (0 -25081.66954845155) / Math.Sqrt((73843.08487620296 / (299)) + (16899.548970046642 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.15922131745241758 = (29831.47654559153 - 25081.66954845155) / 29831.47654559153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 31.33234133333334 < 33.27548114358975.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 55.59867456333919 (T) = (0 -31305.1419062604) / Math.Sqrt((177432.9135300424 / (299)) + (103669.80595089601 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11454730078167981 = (35354.95677397184 - 31305.1419062604) / 35354.95677397184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.LinqQuery(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.74641384615384 < 36.69364468571428.
IsChangePoint: Marked as a change because one of 1/6/2023 2:04:15 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 32.981985973214094 (T) = (0 -34905.49433166833) / Math.Sqrt((294584.7940115831 / (299)) + (178904.21963234092 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08275603027196897 = (38054.75476935328 - 34905.49433166833) / 38054.75476935328 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;Int32&gt;.LinqOrderByExtension(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.67483428571428 < 36.412250436813196.
IsChangePoint: Marked as a change because one of 1/6/2023 5:32:27 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 40.58332838879311 (T) = (0 -34934.77786147186) / Math.Sqrt((263286.858829293 / (299)) + (106125.97764062008 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.0806625813621342 = (37999.952088573635 - 34934.77786147186) / 37999.952088573635 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.ContainsTrueComparer<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 37.64 μs 30.41 μs 0.81 0.00 False
ImmutableSortedSet - Duration of single invocation 37.72 μs 33.34 μs 0.88 0.00 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrueComparer<Int32>.SortedSet(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.410661003236243 < 35.26340905629143.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 58.92581861955677 (T) = (0 -30533.26449607834) / Math.Sqrt((197245.08827584476 / (299)) + (255394.99499278617 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1761147263967729 = (37060.09255699208 - 30533.26449607834) / 37060.09255699208 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrueComparer&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.33773809523809 < 35.60359647509266.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 72.5578858344467 (T) = (0 -32742.939104785375) / Math.Sqrt((90310.0255664583 / (299)) + (84680.61765431883 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.12493697532826514 = (37417.80669691573 - 32742.939104785375) / 37417.80669691573 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 41.03 μs 37.16 μs 0.91 0.01 False
ImmutableSortedSet - Duration of single invocation 42.15 μs 36.49 μs 0.87 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 37.15679056754596 < 39.07662562882765.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 41.36812197885888 (T) = (0 -36869.76000015153) / Math.Sqrt((752198.9934455368 / (299)) + (192308.76841529066 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10638003772565663 = (41258.881355240395 - 36869.76000015153) / 41258.881355240395 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 36.49460145578721 < 39.655365546997814.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 40.72026905086423 (T) = (0 -37684.4274264379) / Math.Sqrt((673584.2143392202 / (299)) + (161763.53758866794 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09576245422236819 = (41675.36241157716 - 37684.4274264379) / 41675.36241157716 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedDictionary - Duration of single invocation 35.88 μs 33.25 μs 0.93 0.02 False
ConcurrentDictionary - Duration of single invocation 3.39 μs 2.85 μs 0.84 0.17 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<Int32, Int32>.ImmutableSortedDictionary(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.248214865303474 < 34.210788704932284.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 1/3/2023 6:29:42 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 45.97290585418548 (T) = (0 -32528.4084782062) / Math.Sqrt((599265.5830427385 / (299)) + (60005.86671170876 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08860490006096818 = (35690.787102522496 - 32528.4084782062) / 35690.787102522496 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 2.851367267585719 < 3.2646418253095058.
IsChangePoint: Marked as a change because one of 2/9/2023 10:24:28 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 35.46627225546193 (T) = (0 -2994.1393419712854) / Math.Sqrt((14563.036970311723 / (299)) + (2563.2196432119035 / (19))) is greater than 1.9674995188240387 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (19) - 2, .975) and 0.1383077123495631 = (3474.719902780329 - 2994.1393419712854) / 3474.719902780329 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BenchStringSort - Duration of single invocation 1.62 secs 1.48 secs 0.91 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchStringSort

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.475355430769231 < 1.5444430851666666.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 47.143872220420455 (T) = (0 -1484465653.093573) / Math.Sqrt((394391236171608.2 / (299)) + (154497683648081.5 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08401726307795344 = (1620626233.7233398 - 1484465653.093573) / 1620626233.7233398 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsSuffix_SecondHalf - Duration of single invocation 225.04 ns 168.88 ns 0.75 0.01 False
IsSuffix_SecondHalf - Duration of single invocation 225.26 ns 169.51 ns 0.75 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False))

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 168.87775084692794 < 214.42890285815014.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 58.059279279359124 (T) = (0 -169.2166074008927) / Math.Sqrt((168.50232773484998 / (299)) + (0.07510610127869334 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.2053081722831122 = (212.93361967373448 - 169.2166074008927) / 212.93361967373448 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False))

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 169.51037410084427 < 213.9863010797195.
IsChangePoint: Marked as a change because one of 1/13/2023 4:18:46 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 57.42511039162931 (T) = (0 -169.50371135691512) / Math.Sqrt((167.72650701156434 / (299)) + (0.2510647938285487 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.20401647274626583 = (212.94876784916522 - 169.50371135691512) / 212.94876784916522 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.Sort<IntClass>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 46.98 μs 42.19 μs 0.90 0.03 False
LinqOrderByExtension - Duration of single invocation 73.85 μs 67.40 μs 0.91 0.00 False
LinqQuery - Duration of single invocation 73.70 μs 67.63 μs 0.92 0.01 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;IntClass&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntClass>.Array(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 42.190842666666676 < 44.0014320595238.
IsChangePoint: Marked as a change because one of 1/10/2023 11:45:30 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 14.28141331417643 (T) = (0 -42512.58866217117) / Math.Sqrt((1009728.4890341564 / (299)) + (934287.4812371021 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.06710150649270144 = (45570.433394465086 - 42512.58866217117) / 45570.433394465086 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntClass&gt;.LinqOrderByExtension(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 67.40393428571427 < 69.50219067857144.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 31.062140891809914 (T) = (0 -68151.63081901432) / Math.Sqrt((322359.72798216785 / (299)) + (596073.417320185 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.0710644541543325 = (73365.295497409 - 68151.63081901432) / 73365.295497409 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntClass&gt;.LinqQuery(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 67.6324 < 69.80613232142855.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 29.53962063462827 (T) = (0 -67836.2011138861) / Math.Sqrt((299025.0244768998 / (299)) + (728900.0730377621 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.07445933682527478 = (73293.59347778311 - 67836.2011138861) / 73293.59347778311 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.Sort<IntStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqQuery - Duration of single invocation 37.51 μs 34.09 μs 0.91 0.01 False
Array_Comparison - Duration of single invocation 27.27 μs 24.36 μs 0.89 0.01 False
LinqOrderByExtension - Duration of single invocation 37.84 μs 33.49 μs 0.89 0.00 False
Array_ComparerClass - Duration of single invocation 27.09 μs 24.54 μs 0.91 0.01 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;IntStruct&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntStruct>.LinqQuery(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.09353076923077 < 35.59067362912088.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 55.7924896790156 (T) = (0 -33570.976901098904) / Math.Sqrt((177109.56228388046 / (299)) + (105078.71158209213 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1085525083363855 = (37658.950431784746 - 33570.976901098904) / 37658.950431784746 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.Array_Comparison(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 24.36197166666667 < 25.59386214761905.
IsChangePoint: Marked as a change because one of 12/10/2022 11:25:25 PM, 1/5/2023 12:24:13 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 27.77528082137825 (T) = (0 -24199.83656876457) / Math.Sqrt((1837754.2171832512 / (299)) + (279330.2997915835 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.13610766800501828 = (28012.560908927186 - 24199.83656876457) / 28012.560908927186 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.LinqOrderByExtension(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.489158571428575 < 35.74504675.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 47.692834641022564 (T) = (0 -33704.4946913087) / Math.Sqrt((137458.12756905027 / (299)) + (143922.97137983426 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10586873144246986 = (37695.24216022883 - 33704.4946913087) / 37695.24216022883 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;IntStruct&gt;.Array_ComparerClass(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 24.542057333333336 < 25.81768747179487.
IsChangePoint: Marked as a change because one of 12/10/2022 11:25:25 PM, 1/5/2023 12:24:13 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 27.385488144726626 (T) = (0 -24258.99681152181) / Math.Sqrt((1868893.214355754 / (299)) + (289491.8690346348 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.13589892343152452 = (28074.25828915677 - 24258.99681152181) / 28074.25828915677 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in IfStatements.IfStatements

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Single - Duration of single invocation 69.91 μs 28.08 μs 0.40 0.04 False
SingleArray - Duration of single invocation 80.77 μs 34.38 μs 0.43 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'IfStatements.IfStatements*'

Payloads

Baseline
Compare

Histogram

IfStatements.IfStatements.Single

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 28.079059694793532 < 66.64833492784079.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 83.628154041665 (T) = (0 -33772.41559547713) / Math.Sqrt((5620991.072180489 / (299)) + (3399412.4973660833 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5076050155340466 = (68588.05767915436 - 33772.41559547713) / 68588.05767915436 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### IfStatements.IfStatements.SingleArray

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.38298530549111 < 76.41984619680149.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 283.3353954438043 (T) = (0 -34806.17307918566) / Math.Sqrt((4594092.657421927 / (299)) + (247994.68761947006 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5705550296458218 = (81049.20416341074 - 34806.17307918566) / 81049.20416341074 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Linq.Tests.Perf_OrderBy

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OrderByValueType - Duration of single invocation 42.51 μs 38.51 μs 0.91 0.06 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_OrderBy*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_OrderBy.OrderByValueType(NumberOfPeople: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 38.51276939655173 < 39.93101530211141.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 19.132749611201643 (T) = (0 -38411.979589523056) / Math.Sqrt((448459.931654579 / (299)) + (535850.4860140091 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.07415423261068496 = (41488.52966929528 - 38411.979589523056) / 41488.52966929528 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 154.12 μs 140.10 μs 0.91 0.00 False
ConcurrentDictionary - Duration of single invocation 60.42 μs 27.03 μs 0.45 0.00 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableDictionary(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 140.10365588803091 < 146.63022199876238.
IsChangePoint: Marked as a change because one of 1/10/2023 7:48:18 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 65.21146415027663 (T) = (0 -140773.65965176394) / Math.Sqrt((8194030.011621963 / (299)) + (654732.8807987688 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09971263802984463 = (156365.25136118772 - 140773.65965176394) / 156365.25136118772 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.027695652173907 < 57.31135233251304.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 782.5113817658126 (T) = (0 -26803.679333150598) / Math.Sqrt((245175.38721193458 / (299)) + (21946.048130992785 / (21))) is greater than 1.967451947860886 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (21) - 2, .975) and 0.5576749959389319 = (60597.2510870085 - 26803.679333150598) / 60597.2510870085 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 199.09 μs 177.14 μs 0.89 0.01 False
ImmutableHashSet - Duration of single invocation 175.60 μs 162.62 μs 0.93 0.06 False
ConcurrentDictionary - Duration of single invocation 63.03 μs 41.89 μs 0.66 0.56 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.ImmutableDictionary(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 177.13753551136367 < 184.0967827139826.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 38.603367156956836 (T) = (0 -178401.92770699438) / Math.Sqrt((25316893.354695424 / (299)) + (3659610.032140041 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09780850650064242 = (197742.86167897837 - 178401.92770699438) / 197742.86167897837 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ImmutableHashSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 162.6224283854167 < 165.8693812385531.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 47.329741281027374 (T) = (0 -163376.84241322675) / Math.Sqrt((13716396.355037335 / (299)) + (1120823.037187657 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08268842118912932 = (178103.97926625478 - 163376.84241322675) / 178103.97926625478 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;String&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 41.89453816621499 < 83.95738915378445.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 46.20892690434907 (T) = (0 -42768.98581274198) / Math.Sqrt((211649540.54153144 / (299)) + (537852.200171241 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.4804053559161397 = (82312.21453052401 - 42768.98581274198) / 82312.21453052401 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Numerics.Tests.Perf_BigInteger

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToByteArray - Duration of single invocation 16.27 ns 13.91 ns 0.86 0.03 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_BigInteger*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BigInteger.ToByteArray(numberString: 123)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 13.914122218722085 < 15.406046655876338.
IsChangePoint: Marked as a change because one of 1/27/2023 7:12:54 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 16.703725093477722 (T) = (0 -14.65938384418851) / Math.Sqrt((0.12037531270651858 / (299)) + (0.1269676443610196 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08217427225556946 = (15.971859799805513 - 14.65938384418851) / 15.971859799805513 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Tests.Perf_Boolean

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 1.73 ns 0.61 ns 0.35 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Boolean*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Boolean.ToString(value: False)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 0.6101851114456653 < 1.625850057703353.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 231.22998865574513 (T) = (0 -0.6081970497868993) / Math.Sqrt((0.004824264248088442 / (299)) + (0.0001375023795688759 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.6427004731649961 = (1.702205024379325 - 0.6081970497868993) / 1.702205024379325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dequeue_And_Enqueue - Duration of single invocation 16.21 μs 15.17 μs 0.94 0.34 False
Dequeue_And_Enqueue - Duration of single invocation 328.66 μs 288.80 μs 0.88 0.02 False
HeapSort - Duration of single invocation 5.27 μs 4.69 μs 0.89 0.05 False
HeapSort - Duration of single invocation 134.66 μs 121.29 μs 0.90 0.01 False
Dequeue_And_Enqueue - Duration of single invocation 724.26 ns 662.01 ns 0.91 0.08 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_PriorityQueue<Guid, Guid>.Dequeue_And_Enqueue(Size: 100)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 15.16598732755855 < 15.860691193014066.
IsChangePoint: Marked as a change because one of 12/14/2022 12:26:48 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 15.42573396171309 (T) = (0 -14489.582257306465) / Math.Sqrt((334256.3014000008 / (299)) + (431702.30014468834 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1329388413606468 = (16711.142129863627 - 14489.582257306465) / 16711.142129863627 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;.Dequeue_And_Enqueue(Size: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 288.8022045454545 < 311.91827507617626.
IsChangePoint: Marked as a change because one of 12/14/2022 12:26:48 AM, 1/11/2023 8:19:46 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 48.29752588241019 (T) = (0 -288865.0314157994) / Math.Sqrt((42713697.04919909 / (299)) + (6291951.353679461 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09868696067663316 = (320493.56750974775 - 288865.0314157994) / 320493.56750974775 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;.HeapSort(Size: 100)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.692875163005317 < 5.020636450249062.
IsChangePoint: Marked as a change because one of 12/28/2022 5:13:50 PM, 1/3/2023 6:29:42 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 16.810852737238392 (T) = (0 -4794.4816808356445) / Math.Sqrt((12835.919683478654 / (299)) + (15000.837073807881 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08625384410201703 = (5247.06084932732 - 4794.4816808356445) / 5247.06084932732 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;.HeapSort(Size: 1000)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 121.28629807692306 < 128.10475528223373.
IsChangePoint: Marked as a change because one of 12/10/2022 11:25:25 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 43.54221049398458 (T) = (0 -120119.64123343443) / Math.Sqrt((2998380.922824821 / (299)) + (2079748.4810527163 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10491720318856378 = (134199.4747986868 - 120119.64123343443) / 134199.4747986868 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Tests.Perf_PriorityQueue&lt;Guid, Guid&gt;.Dequeue_And_Enqueue(Size: 10)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 662.0060291504903 < 698.099241769045.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 45.01293327139082 (T) = (0 -673.4765601560739) / Math.Sqrt((253.6180055655602 / (299)) + (36.874723163980306 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09599727073027915 = (744.9939456489556 - 673.4765601560739) / 744.9939456489556 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableList - Duration of single invocation 112.74 μs 99.86 μs 0.89 0.03 False
ImmutableDictionary - Duration of single invocation 345.67 μs 326.49 μs 0.94 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<String>.ImmutableList(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 99.85636733058608 < 106.82760816510165.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 38.253834141292046 (T) = (0 -99857.9736528757) / Math.Sqrt((3579391.2671758127 / (299)) + (2344892.0966240605 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11653243502263454 = (113029.58661015934 - 99857.9736528757) / 113029.58661015934 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CreateAddAndClear&lt;String&gt;.ImmutableDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 326.49137834821425 < 329.0685336999654.
IsChangePoint: Marked as a change because one of 1/11/2023 2:37:52 AM, 2/6/2023 4:11:28 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 23.34698713488475 (T) = (0 -328745.9178691631) / Math.Sqrt((247102037.68986994 / (299)) + (42421839.31349082 / (37))) is greater than 1.9670919629190173 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (37) - 2, .975) and 0.0907056982998162 = (361539.6217203597 - 328745.9178691631) / 361539.6217203597 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.Sort<BigStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqOrderByExtension - Duration of single invocation 47.50 μs 42.94 μs 0.90 0.01 False
LinqQuery - Duration of single invocation 47.13 μs 42.84 μs 0.91 0.00 False
Array_ComparerClass - Duration of single invocation 36.95 μs 29.76 μs 0.81 0.28 False
Array_ComparerStruct - Duration of single invocation 43.23 μs 35.99 μs 0.83 0.17 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<BigStruct>.LinqOrderByExtension(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 42.93692533333333 < 44.926054664285715.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 58.63909039177262 (T) = (0 -42848.53136829837) / Math.Sqrt((261246.33649103774 / (299)) + (115173.75873705951 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09662731334956766 = (47431.732220258025 - 42848.53136829837) / 47431.732220258025 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.LinqQuery(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 42.844258571428576 < 45.00537726666667.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 48.44006586654838 (T) = (0 -42912.253255411255) / Math.Sqrt((258052.45919425884 / (299)) + (177472.45070492924 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09638954438854791 = (47489.77060737255 - 42912.253255411255) / 47489.77060737255 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.Array_ComparerClass(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 29.764678461538466 < 35.16079770000001.
IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 10.268021029818827 (T) = (0 -32495.317283356875) / Math.Sqrt((7621187.45375209 / (299)) + (9800024.989058774 / (38))) is greater than 1.967070609662329 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (38) - 2, .975) and 0.143990107607548 = (37961.38055430188 - 32495.317283356875) / 37961.38055430188 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.Sort&lt;BigStruct&gt;.Array_ComparerStruct(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 35.98511428571429 < 41.181956673076925.
IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 25.12054850490019 (T) = (0 -37558.47954409176) / Math.Sqrt((4906804.643137079 / (299)) + (1633240.8837016136 / (38))) is greater than 1.967070609662329 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (38) - 2, .975) and 0.14015279734265684 = (43680.41138939327 - 37558.47954409176) / 43680.41138939327 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedSet - Duration of single invocation 36.39 μs 30.88 μs 0.85 0.00 False
ImmutableSortedSet - Duration of single invocation 35.46 μs 31.09 μs 0.88 0.00 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.SortedSet(Size: 512)

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.876029339250497 < 34.44429429407457.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 56.616885162824026 (T) = (0 -31367.217069987462) / Math.Sqrt((480898.1236598248 / (299)) + (111721.04058622166 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.12861257290609202 = (35996.86671472604 - 31367.217069987462) / 35996.86671472604 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ImmutableSortedSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 31.09035838293651 < 33.67907047788427.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 73.48273560151628 (T) = (0 -29748.409785150263) / Math.Sqrt((373667.0938601396 / (299)) + (130503.7729877598 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.17309719503003182 = (35975.70307701481 - 29748.409785150263) / 35975.70307701481 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Tests.Perf_Random

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NextBytes_span - Duration of single invocation 8.13 μs 3.26 μs 0.40 0.03 False 37284.680227231074 40175.134719014626 1.0775239179783147) Trace Trace
Next_long - Duration of single invocation 48.94 ns 31.58 ns 0.65 0.07 False
Next - Duration of single invocation 10.55 ns 3.74 ns 0.35 0.07 False
Next_int_int - Duration of single invocation 12.99 ns 6.37 ns 0.49 0.12 False
Next_long_long - Duration of single invocation 53.45 ns 36.21 ns 0.68 0.07 False
Next_int - Duration of single invocation 12.86 ns 6.36 ns 0.49 0.13 False
NextBytes - Duration of single invocation 8.09 μs 3.19 μs 0.39 0.01 False
NextDouble - Duration of single invocation 11.53 ns 4.76 ns 0.41 0.09 False
NextSingle - Duration of single invocation 11.58 ns 4.94 ns 0.43 0.12 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Random*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Random.NextBytes_span

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.2571147555096416 < 7.719729305175932.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 296.3357572386153 (T) = (0 -3303.378573757675) / Math.Sqrt((48746.574935253775 / (299)) + (1985.060073974227 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5880728709550693 = (8019.327548094947 - 3303.378573757675) / 8019.327548094947 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Tests.Perf_Random.NextBytes_span()
       push      rdi
       push      rsi
       push      rbx
       sub       rsp,30
       xor       eax,eax
       mov       [rsp+20],rax
       mov       rsi,[rcx+10]
       mov       rcx,[rcx+18]
       test      rcx,rcx
       je        short M00_L01
       lea       rdi,[rcx+10]
       mov       ebx,[rcx+8]
M00_L00:
       mov       rcx,rsi
       mov       [rsp+20],rdi
       mov       [rsp+28],ebx
       lea       rdx,[rsp+20]
       mov       rax,[rcx]
       mov       rax,[rax+48]
       call      qword ptr [rax+28]
       nop
       add       rsp,30
       pop       rbx
       pop       rsi
       pop       rdi
       ret
M00_L01:
       xor       edi,edi
       xor       ebx,ebx
       jmp       short M00_L00
; Total bytes of code 76

Compare Jit Disasm

; System.Tests.Perf_Random.NextBytes_span()
       push      rdi
       push      rsi
       push      rbx
       sub       rsp,30
       xor       eax,eax
       mov       [rsp+20],rax
       mov       rsi,[rcx+10]
       mov       rcx,[rcx+18]
       test      rcx,rcx
       je        short M00_L01
       lea       rdi,[rcx+10]
       mov       ebx,[rcx+8]
M00_L00:
       mov       rcx,rsi
       mov       [rsp+20],rdi
       mov       [rsp+28],ebx
       lea       rdx,[rsp+20]
       mov       rax,[rcx]
       mov       rax,[rax+48]
       call      qword ptr [rax+28]
       nop
       add       rsp,30
       pop       rbx
       pop       rsi
       pop       rdi
       ret
M00_L01:
       xor       edi,edi
       xor       ebx,ebx
       jmp       short M00_L00
; Total bytes of code 76

System.Tests.Perf_Random.Next_long

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 31.576064577632735 < 45.83966090281622.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 92.4902664846023 (T) = (0 -31.30474486212105) / Math.Sqrt((1.4032183077500195 / (299)) + (0.6429243007114432 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.3523814421660516 = (48.338245535804575 - 31.30474486212105) / 48.338245535804575 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Tests.Perf_Random.Next_long()
       mov       rcx,[rcx+10]
       mov       edx,16
       mov       rax,[rcx]
       mov       rax,[rax+48]
       jmp       qword ptr [rax]
; Total bytes of code 19

System.Tests.Perf_Random.Next

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.7420181261536243 < 10.0217335735543.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 405.5612218496521 (T) = (0 -3.7039747840206654) / Math.Sqrt((0.055447080581758804 / (299)) + (0.002497259589007184 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.6543587005029213 = (10.716239029913645 - 3.7039747840206654) / 10.716239029913645 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Tests.Perf_Random.Next()
       mov       rcx,[rcx+10]
       mov       rax,[rcx]
       mov       rax,[rax+40]
       jmp       qword ptr [rax+20]
; Total bytes of code 15

System.Tests.Perf_Random.Next_int_int

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.3706531965393705 < 12.369090498862642.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 285.92828454588533 (T) = (0 -6.4450013763457745) / Math.Sqrt((0.13985614051371534 / (299)) + (0.0016191555269480025 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5079264890410757 = (13.097639342109943 - 6.4450013763457745) / 13.097639342109943 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.Next_long_long

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 36.20522826370482 < 50.88984823438968.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 90.30331149380063 (T) = (0 -36.60810446575701) / Math.Sqrt((1.66511619662198 / (299)) + (0.6589998581833985 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.3173718673420485 = (53.62818013845387 - 36.60810446575701) / 53.62818013845387 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.Next_int

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 6.360947968316809 < 12.247440460445711.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 173.99892243491877 (T) = (0 -6.487500218538454) / Math.Sqrt((0.14714495035163233 / (299)) + (0.019441662510273353 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.49870619800712646 = (12.94151292664632 - 6.487500218538454) / 12.94151292664632 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.NextBytes

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 3.1942959582790094 < 7.718167781079722.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 416.44444992784645 (T) = (0 -3247.3581979770884) / Math.Sqrt((33400.84878039226 / (299)) + (373.3547246868211 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5926224724544326 = (7971.372936406888 - 3247.3581979770884) / 7971.372936406888 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.NextDouble

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.75560870014414 < 10.781480581421762.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 135.9584694000711 (T) = (0 -4.8175076074102465) / Math.Sqrt((0.060045009412680664 / (299)) + (0.048145928120051404 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.5797407444192517 = (11.463180271313773 - 4.8175076074102465) / 11.463180271313773 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Random.NextSingle

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 4.941569587766499 < 10.883522475269142.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 120.82359095221976 (T) = (0 -4.767891242044872) / Math.Sqrt((0.055286609525956966 / (299)) + (0.06519921801981896 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.587105835577887 = (11.54749001773376 - 4.767891242044872) / 11.54749001773376 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@EgorBo EgorBo mentioned this issue Mar 28, 2023
Merged
3 tasks
@EgorBo EgorBo closed this as completed Mar 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EgorBo EgorBo

Labels
arch-x64 branch-refs/heads/main kind-micro os-windows perf-improvement runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@EgorBo