Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions from using reloadWeight when evaluating spill cost #54352

Closed
DrewScoggins opened this issue Jun 17, 2021 · 4 comments · Fixed by #55247
Closed

Regressions from using reloadWeight when evaluating spill cost #54352

DrewScoggins opened this issue Jun 17, 2021 · 4 comments · Fixed by #55247
Assignees
@kunalspathak
Labels
arch-x64 arch-x86 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
6.0.0

Comments

@DrewScoggins
Copy link
Member

Run Information

Architecture x64
OS ubuntu 18.04
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in BenchmarksGame.FannkuchRedux_5

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 23.23 ms 25.62 ms 1.10 0.01

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'BenchmarksGame.FannkuchRedux_5*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@DrewScoggins DrewScoggins added os-linux Linux OS (any supported distro) tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark arch-x64 labels Jun 17, 2021
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jun 17, 2021
@DrewScoggins
Copy link
Member Author

#53853

@DrewScoggins DrewScoggins changed the title Regressions in BenchmarksGame.FannkuchRedux_5 Regressions from using reloadWeight when evaluating spill cost Jun 17, 2021
@DrewScoggins
Copy link
Member Author

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in SIMD.ConsoleMandel

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
VectorDoubleSinglethreadRaw - Duration of single invocation 343.96 ms 485.19 ms 1.41 0.00
VectorFloatSinglethreadRaw - Duration of single invocation 198.65 ms 262.33 ms 1.32 0.00
VectorDoubleSinglethreadADT - Duration of single invocation 343.01 ms 433.14 ms 1.26 0.00

graph
graph
graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'SIMD.ConsoleMandel*'

Payloads

Baseline
Compare

Histogram

SIMD.ConsoleMandel.VectorDoubleSinglethreadRaw

SIMD.ConsoleMandel.VectorFloatSinglethreadRaw

SIMD.ConsoleMandel.VectorDoubleSinglethreadADT

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToLowerInvariant - Duration of single invocation 19.97 ns 21.90 ns 1.10 0.01
ToUpperInvariant - Duration of single invocation 21.15 ns 22.35 ns 1.06 0.02
ToLowerInvariant - Duration of single invocation 19.99 ns 22.03 ns 1.10 0.02
ToUpperInvariant - Duration of single invocation 21.08 ns 22.34 ns 1.06 0.03

graph
graph
graph
graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

System.Tests.Perf_String.ToUpperInvariant(s: "TeSt")

System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

System.Tests.Perf_String.ToUpperInvariant(s: "test")

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 11.73 μs 15.14 μs 1.29 0.34

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.Dictionary(Size: 512)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateUsingIndexer - Duration of single invocation 2.42 μs 2.88 μs 1.19 0.00
EnumerateUsingIndexer - Duration of single invocation 2.42 μs 2.88 μs 1.19 0.01

graph
graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in BenchmarksGame.KNucleotide_9

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
RunBench - Duration of single invocation 97.83 ms 106.06 ms 1.08 0.27

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'BenchmarksGame.KNucleotide_9*'

Payloads

Baseline
Compare

Histogram

BenchmarksGame.KNucleotide_9.RunBench

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Memory.ReadOnlySequence

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Slice_Start_And_EndPosition - Duration of single invocation 21.03 ns 26.84 ns 1.28 0.10

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Memory.ReadOnlySequence*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySequence.Slice_Start_And_EndPosition(Segment: Multiple)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 12.75 ns 13.83 ns 1.08 0.11

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.Parse(value: "255")

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.IO.Tests.Perf_StreamWriter

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteCharArray - Duration of single invocation 120.85 ms 137.98 ms 1.14 0.05
WriteString - Duration of single invocation 120.85 ms 137.90 ms 1.14 0.05

graph
graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.IO.Tests.Perf_StreamWriter*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 2)

System.IO.Tests.Perf_StreamWriter.WriteString(writeLength: 2)

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x86
OS Windows 10.0.18362
Baseline d07f9111c57fab97aa135bebb2687471a8bfc5bf
Compare 75ede83214700ba49b70c0ac746c3541ef219a91
Diff Diff

Regressions in System.Memory.Slice<Byte>

Benchmark Baseline Test Test/Base Test Quality Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MemoryStartSpan - Duration of single invocation 2.83 ns 4.96 ns 1.75 0.07

graph
Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Memory.Slice&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

System.Memory.Slice<Byte>.MemoryStartSpan

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kunalspathak kunalspathak self-assigned this Jun 18, 2021
@kunalspathak kunalspathak added this to the 6.0.0 milestone Jun 18, 2021
@kunalspathak
Copy link
Member

x64 improvements: DrewScoggins/performance-2#6592 and DrewScoggins/performance-2#6593

x86 improvements: DrewScoggins/performance-2#6612

@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 18, 2021
@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Jun 21, 2021
@kunalspathak kunalspathak mentioned this issue Jul 7, 2021
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Aug 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kunalspathak kunalspathak

Labels
arch-x64 arch-x86 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

Use correct assignedInterval for SPILL_COST calculation kunalspathak/runtime
4 participants
@DrewScoggins @jeffschwMSFT @kunalspathak @JulieLeeMSFT