Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.IO.Tests.Perf_Path #75548

Closed
performanceautofiler bot opened this issue Sep 13, 2022 · 4 comments
Closed

Regressions in System.IO.Tests.Perf_Path #75548

performanceautofiler bot opened this issue Sep 13, 2022 · 4 comments
Assignees
@EgorBo
Labels
tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Sep 13, 2022 •
edited by EgorBo
Loading

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetPathRoot - Duration of single invocation 28.73 ns 30.69 ns 1.07 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetPathRoot

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.686773278953897 > 30.230685990506515.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/9/2022 3:12:19 AM, 9/13/2022 2:30:06 AM falls between 9/4/2022 8:45:57 AM and 9/13/2022 2:30:06 AM.
IsRegressionStdDev: Marked as regression because -20.65085902998708 (T) = (0 -31.036276806922494) / Math.Sqrt((0.16816385039880005 / (40)) + (0.14906292245135205 / (17))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (17) - 2, .025) and -0.08199873573093718 = (28.684208014306172 - 31.036276806922494) / 28.684208014306172 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.IO.Tests.Perf_Path.GetPathRoot()
       push      rbp
       sub       rsp,20
       lea       rbp,[rsp+20]
       mov       [rbp+10],rcx
       mov       rcx,[rbp+10]
       mov       rcx,[rcx+8]
       call      qword ptr [System.IO.Path.GetPathRoot(System.String)]
       nop
       add       rsp,20
       pop       rbp
       ret
       add       [rcx],bl
       add       eax,32050002
       add       [rax],edx
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax+65],ch
       (bad)
; Total bytes of code 60

; System.IO.Path.GetPathRoot(System.String)
       push      rsi
       sub       rsp,40
       xorps     xmm4,xmm4
       movaps    [rsp+20],xmm4
       movaps    [rsp+30],xmm4
       mov       rsi,rcx
       test      rsi,rsi
       je        short M01_L03
       lea       rcx,[rsi+0C]
       mov       eax,[rsi+8]
M01_L00:
       mov       [rsp+20],rcx
       mov       [rsp+28],eax
       lea       rcx,[rsp+20]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       test      eax,eax
       jne       short M01_L05
       test      rsi,rsi
       je        short M01_L02
       lea       rdx,[rsi+0C]
       mov       ecx,[rsi+8]
M01_L01:
       mov       [rsp+20],rdx
       mov       [rsp+28],ecx
       lea       rdx,[rsp+20]
       lea       rcx,[rsp+30]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       mov       ecx,[rsi+8]
       cmp       ecx,[rsp+38]
       je        short M01_L04
       lea       rcx,[rsp+30]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       mov       rcx,rax
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       nop
       add       rsp,40
       pop       rsi
       ret
M01_L02:
       xor       edx,edx
       xor       ecx,ecx
       jmp       short M01_L01
M01_L03:
       xor       ecx,ecx
       xor       eax,eax
       jmp       short M01_L00
M01_L04:
       mov       rcx,rsi
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       nop
       add       rsp,40
       pop       rsi
       ret
M01_L05:
       xor       eax,eax
       add       rsp,40
       pop       rsi
       ret
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       push      rdi
       push      rsi
       sub       rsp,38
       mov       rdi,rcx
       mov       rsi,rdx
       movups    xmm0,[rsi]
       (bad)
; Total bytes of code 192

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 42.20 μs 53.04 μs 1.26 0.10 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.03504347826087 > 43.573991015625005.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/9/2022 3:12:19 AM, 9/13/2022 2:30:06 AM falls between 9/4/2022 8:45:57 AM and 9/13/2022 2:30:06 AM.
IsRegressionStdDev: Marked as regression because -14.136194439760997 (T) = (0 -60375.9271009913) / Math.Sqrt((511739.84831417294 / (41)) + (30211974.304730527 / (17))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (17) - 2, .025) and -0.45607654023901484 = (41464.80314220336 - 60375.9271009913) / 41464.80314220336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Text.Perf_Utf8Encoding.GetString()
       push      rbp
       sub       rsp,30
       lea       rbp,[rsp+30]
       xor       eax,eax
       mov       [rbp-8],rax
       mov       [rbp+10],rcx
       mov       rcx,[rbp+10]
       mov       rcx,[rcx+28]
       mov       [rbp-8],rcx
       mov       rcx,[rbp-8]
       mov       rdx,[rbp+10]
       mov       rdx,[rdx+20]
       mov       rax,[rbp-8]
       mov       rax,[rax]
       mov       rax,[rax+68]
       call      qword ptr [rax+28]
       nop
       add       rsp,30
       pop       rbp
       ret
       add       [rax],al
       add       [rcx],bl
       add       eax,52050002
       add       [rax],edx
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],dh
       (bad)
; Total bytes of code 90

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Sep 13, 2022
@EgorBo EgorBo changed the title [Perf] Windows/x64: 4 Regressions on 9/9/2022 1:01:45 PM Regressions in System.IO.Tests.Perf_Path Sep 13, 2022
@EgorBo EgorBo removed refs/heads/main untriaged New issue has not been triaged by the area owner labels Sep 13, 2022
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues Sep 13, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 13, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Sep 13, 2022
@ghost
Copy link

ghost commented Sep 13, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetPathRoot - Duration of single invocation 28.73 ns 30.69 ns 1.07 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetPathRoot

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.686773278953897 > 30.230685990506515.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/9/2022 3:12:19 AM, 9/13/2022 2:30:06 AM falls between 9/4/2022 8:45:57 AM and 9/13/2022 2:30:06 AM.
IsRegressionStdDev: Marked as regression because -20.65085902998708 (T) = (0 -31.036276806922494) / Math.Sqrt((0.16816385039880005 / (40)) + (0.14906292245135205 / (17))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (40) + (17) - 2, .025) and -0.08199873573093718 = (28.684208014306172 - 31.036276806922494) / 28.684208014306172 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.IO.Tests.Perf_Path.GetPathRoot()
       push      rbp
       sub       rsp,20
       lea       rbp,[rsp+20]
       mov       [rbp+10],rcx
       mov       rcx,[rbp+10]
       mov       rcx,[rcx+8]
       call      qword ptr [System.IO.Path.GetPathRoot(System.String)]
       nop
       add       rsp,20
       pop       rbp
       ret
       add       [rcx],bl
       add       eax,32050002
       add       [rax],edx
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax+65],ch
       (bad)
; Total bytes of code 60

; System.IO.Path.GetPathRoot(System.String)
       push      rsi
       sub       rsp,40
       xorps     xmm4,xmm4
       movaps    [rsp+20],xmm4
       movaps    [rsp+30],xmm4
       mov       rsi,rcx
       test      rsi,rsi
       je        short M01_L03
       lea       rcx,[rsi+0C]
       mov       eax,[rsi+8]
M01_L00:
       mov       [rsp+20],rcx
       mov       [rsp+28],eax
       lea       rcx,[rsp+20]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       test      eax,eax
       jne       short M01_L05
       test      rsi,rsi
       je        short M01_L02
       lea       rdx,[rsi+0C]
       mov       ecx,[rsi+8]
M01_L01:
       mov       [rsp+20],rdx
       mov       [rsp+28],ecx
       lea       rdx,[rsp+20]
       lea       rcx,[rsp+30]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       mov       ecx,[rsi+8]
       cmp       ecx,[rsp+38]
       je        short M01_L04
       lea       rcx,[rsp+30]
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       mov       rcx,rax
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       nop
       add       rsp,40
       pop       rsi
       ret
M01_L02:
       xor       edx,edx
       xor       ecx,ecx
       jmp       short M01_L01
M01_L03:
       xor       ecx,ecx
       xor       eax,eax
       jmp       short M01_L00
M01_L04:
       mov       rcx,rsi
       call      qword ptr [System.Reflection.CustomAttributeExtensions.GetCustomAttribute[[System.__Canon, System.Private.CoreLib]](System.Reflection.Assembly)]
       nop
       add       rsp,40
       pop       rsi
       ret
M01_L05:
       xor       eax,eax
       add       rsp,40
       pop       rsi
       ret
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       int       3
       push      rdi
       push      rsi
       sub       rsp,38
       mov       rdi,rcx
       mov       rsi,rdx
       movups    xmm0,[rsi]
       (bad)
; Total bytes of code 192

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 42.20 μs 53.04 μs 1.26 0.10 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 53.03504347826087 > 43.573991015625005.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/9/2022 3:12:19 AM, 9/13/2022 2:30:06 AM falls between 9/4/2022 8:45:57 AM and 9/13/2022 2:30:06 AM.
IsRegressionStdDev: Marked as regression because -14.136194439760997 (T) = (0 -60375.9271009913) / Math.Sqrt((511739.84831417294 / (41)) + (30211974.304730527 / (17))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (17) - 2, .025) and -0.45607654023901484 = (41464.80314220336 - 60375.9271009913) / 41464.80314220336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```### Baseline Jit Disasm

```assembly
; System.Text.Perf_Utf8Encoding.GetString()
       push      rbp
       sub       rsp,30
       lea       rbp,[rsp+30]
       xor       eax,eax
       mov       [rbp-8],rax
       mov       [rbp+10],rcx
       mov       rcx,[rbp+10]
       mov       rcx,[rcx+28]
       mov       [rbp-8],rcx
       mov       rcx,[rbp-8]
       mov       rdx,[rbp+10]
       mov       rdx,[rdx+20]
       mov       rax,[rbp-8]
       mov       rax,[rax]
       mov       rax,[rax+68]
       call      qword ptr [rax+28]
       nop
       add       rsp,30
       pop       rbp
       ret
       add       [rax],al
       add       [rcx],bl
       add       eax,52050002
       add       [rax],edx
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],al
       add       [rax],dh
       (bad)
; Total bytes of code 90

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 7.10 μs 9.87 μs 1.39 0.41 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.873726901686402 > 8.311290985129968.
IsChangePoint: Marked as a change because one of 8/12/2022 4:09:16 AM, 8/16/2022 10:54:40 PM, 9/9/2022 3:12:19 AM, 9/13/2022 2:30:06 AM falls between 9/4/2022 8:45:57 AM and 9/13/2022 2:30:06 AM.
IsRegressionStdDev: Marked as regression because -10.600310142851017 (T) = (0 -8515.773074339508) / Math.Sqrt((478968.1463606895 / (41)) + (144266.09401836756 / (17))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (41) + (17) - 2, .025) and -0.2147402997815578 = (7010.365158602927 - 8515.773074339508) / 7010.365158602927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline bc183b8208b7a2b3310715ad2181d6a4720a159c
Compare 071fe7ce868a63113143b949c3c4185f9e5578cf
Diff Diff
Author: performanceautofiler[bot]
Assignees: EgorBo
Labels:

area-CodeGen-coreclr
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Sep 13, 2022

PerfPath regression is most likely #75323 cc @stephentoub
System.Text.Perf_Utf8Encoding regression is #73055 cc @adamsitnik

@EgorBo EgorBo added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark and removed area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI untriaged New issue has not been triaged by the area owner labels Sep 13, 2022
@stephentoub
Copy link
Member

PerfPath regression is most likely #75323 cc @stephentoub

That PR was for Uri. I don't see how that would impact Path.GetPathRoot.

@EgorBo EgorBo mentioned this issue Sep 14, 2022
Closed
@EgorBo
Copy link
Member

EgorBo commented Sep 14, 2022

Ah, closing in favor of #75544 then

@EgorBo EgorBo closed this as completed Sep 14, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Oct 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@EgorBo EgorBo

Labels
tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@EgorBo @stephentoub