Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338

Closed
performanceautofiler bot opened this issue Apr 20, 2022 · 14 comments
Closed

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests #68338

performanceautofiler bot opened this issue Apr 20, 2022 · 14 comments
Labels
arch-x64 area-System.Buffers runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 5c57f2c0cda44176e237574ceb51d659ef9915fa
Compare f216e77788933d4d6906f7f91d09375719f4e6b2
Diff Diff

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadStructFieldByFieldUsingBitConverterBE - Duration of single invocation 21.41 ns 32.53 ns 1.52 0.11 True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.53420517559187 > 21.87106496083853.
IsChangePoint: Marked as a change because one of 4/13/2022 1:59:07 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -88.23270965365063 (T) = (0 -32.283158821426134) / Math.Sqrt((0.3468327710995198 / (30)) + (0.1563465723247371 / (26))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (26) - 2, .025) and -0.5681883882104664 = (20.586275899074835 - 32.283158821426134) / 20.586275899074835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Apr 20, 2022
@kunalspathak
Copy link
Member

kunalspathak commented Apr 21, 2022 •
edited

This is a clear regression but the range e0560ba...12dd2ab doesn't have any related commit.

@kunalspathak
Copy link
Member

There was also an arcade update of dotnet/arcade@d2715c6...2e24ed0 in performance repo and it might be one of the cause.

image

@kunalspathak kunalspathak changed the title [Perf] Changes at 4/13/2022 5:30:39 PM Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests Apr 21, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Apr 21, 2022
@AndyAyersMS AndyAyersMS mentioned this issue Apr 22, 2022
Merged
5 tasks
@ghost
Copy link

ghost commented Apr 22, 2022

Tagging subscribers to this area: @dotnet/area-system-buffers
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 5c57f2c0cda44176e237574ceb51d659ef9915fa
Compare f216e77788933d4d6906f7f91d09375719f4e6b2
Diff Diff

Regressions in System.Buffers.Binary.Tests.BinaryReadAndWriteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadStructFieldByFieldUsingBitConverterBE - Duration of single invocation 21.41 ns 32.53 ns 1.52 0.11 True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Buffers.Binary.Tests.BinaryReadAndWriteTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.53420517559187 > 21.87106496083853.
IsChangePoint: Marked as a change because one of 4/13/2022 1:59:07 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -88.23270965365063 (T) = (0 -32.283158821426134) / Math.Sqrt((0.3468327710995198 / (30)) + (0.1563465723247371 / (26))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (26) - 2, .025) and -0.5681883882104664 = (20.586275899074835 - 32.283158821426134) / 20.586275899074835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Buffers, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.18362, Regression, CoreClr, x64
Milestone: -

@DrewScoggins DrewScoggins added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Apr 22, 2022
@kunalspathak
Copy link
Member

More regressions in arm64: dotnet/perf-autofiling-issues#4727

@B1Z0N B1Z0N mentioned this issue Apr 28, 2022
Closed
@AndyAyersMS
Copy link
Member

Another regression (windows x64): dotnet/perf-autofiling-issues#4715

@danmoseley
Copy link
Member

Prompted by @kunalspathak observation above, in the perf repo over this period was
https://github.com/dotnet/performance/compare/c41c5829c75cc0e4d110cdcc2fcef68390003e14..2b90a3bb0e8705c8a

not much of interest there except maybe 2b90a3bb0e8705c8af2a06447657d7c1debde312 which updated Arcade

<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="7.0.0-beta.22211.6">
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="7.0.0-beta.22212.2">
  <Uri>https://github.com/dotnet/arcade</Uri>
  <Sha>d2715c6ef2c3e2847925250f0a8012344fbb9846</Sha>
  <Sha>2e24ed0bdbd4e3b42c296503955c837bb0b08372</Sha>

in Arcade that is
https://github.com/dotnet/arcade/compare/d2715c6ef2c3e28479..2e24ed0bdbd4
which hasn't much either except an update of the Arcade it uses - which brought a new compiler - but how would that affect this?

@AndyAyersMS
Copy link
Member

which brought a new compiler

The actual diff is seemingly: https://github.com/dotnet/arcade/compare/549523c3fc8929da1a3073d1a97f298e0d1dc342..d2715c6ef2c3e28479

which bumped roslyn from 4.2.0-3.22201.5 (4f4a757510f260eb8ac121dd9f8e7706d13751fb) to 4.2.0-4.22208.7 (9a3459303679328da9ccc529888e3c576ce3efec)

I may be reading the diffs wrong, but I don't see anything in the roslyn diffs that looks like it would change IL emission:
dotnet/roslyn@4f4a757...9a34593

We should look at the IL and see if anything's changed.

@dakersnar
Copy link
Contributor

More regressions detected across all configs, for both BE and LE. Here is the data:

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE

Result Ratio Operating System Bit Processor Name
Slower 0.30 ubuntu 20.04 Arm64 Unknown processor
Slower 0.33 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.27 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.44 Windows 11 Arm64 Unknown processor
Slower 0.34 macOS Monterey 12.3 Arm64 Apple M1 Max
Slower 0.47 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower 0.47 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.50 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.55 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.52 Windows 11 X64 AMD Ryzen 9 5950X
Slower 0.49 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.50 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 0.51 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 0.54 pop 22.04 X64 Intel Core i7-6600U CPU 2.60GHz (Skylake)
Slower 0.50 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.45 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.61 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.48 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.48 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.50 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.57 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.61 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Slower 0.50 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE

Result Ratio Operating System Bit Processor Name
Slower 0.40 ubuntu 20.04 Arm64 Unknown processor
Slower 0.49 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.40 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.52 Windows 11 Arm64 Unknown processor
Slower 0.49 macOS Monterey 12.3 Arm64 Apple M1 Max
Slower 0.60 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower 0.83 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.68 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.77 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.66 Windows 11 X64 AMD Ryzen 9 5950X
Slower 0.59 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 1.13 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 0.63 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 0.62 pop 22.04 X64 Intel Core i7-6600U CPU 2.60GHz (Skylake)
Slower 0.72 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.68 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.63 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.63 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.78 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.59 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Slower 0.83 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.66 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Slower 0.65 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

@dakersnar dakersnar mentioned this issue May 16, 2022
Closed
16 tasks
@kunalspathak
Copy link
Member

This benchmark recovered:

image

@danmoseley
Copy link
Member

Any idea why - did it coincide with another arcade update?

@dakersnar
Copy link
Contributor

dakersnar commented May 19, 2022 •
edited

@danmoseley We haven't determined why. I think we should check in on this in a bit, and also monitor the rest of the configs to see if they also recovered. Unfortunately the recovery won't show up in the perf report until preview6 because it seems to have missed the cutoff. EDIT: Nevermind, I thought preview5 snapped already. We will see these changes in the preview5 report.

@kunalspathak
Copy link
Member

kunalspathak commented May 19, 2022 •
edited

Any idea why - did it coincide with another arcade update?

Out of the range 9d972ed...7b5f40f during which performance was improved, we have a theory that #68739 might have impacted, but it is hard to believe that the performance came back to around same point. Also, even if it is because of #68739, it means that we still need to investigate the cause of the initial spike.

@dakersnar
Copy link
Contributor

As expected, preview5 shows the fix to this regression.

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterBE

Result Ratio Operating System Bit Processor Name
Faster 1.27 debian 11 Arm64 Unknown processor
Faster 2.31 ubuntu 18.04 Arm64 Unknown processor
Faster 3.37 ubuntu 20.04 Arm64 Unknown processor
Faster 3.32 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 2.14 macOS Monterey 12.3 Arm64 Apple M1 Max
Faster 1.77 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 1.74 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Faster 1.54 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Faster 1.64 Windows 11 X64 AMD Ryzen 9 5900X
Faster 1.70 Windows 11 X64 AMD Ryzen 9 5950X
Faster 1.82 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 1.56 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Faster 1.76 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Faster 1.65 ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
Faster 1.40 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Faster 1.71 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 1.73 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 1.59 macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

System.Buffers.Binary.Tests.BinaryReadAndWriteTests.ReadStructFieldByFieldUsingBitConverterLE

Result Ratio Operating System Bit Processor Name
Faster 1.43 debian 11 Arm64 Unknown processor
Faster 3.06 ubuntu 18.04 Arm64 Unknown processor
Faster 5.23 ubuntu 20.04 Arm64 Unknown processor
Faster 5.17 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Faster 3.60 macOS Monterey 12.3 Arm64 Apple M1 Max
Faster 2.09 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 2.34 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Faster 2.18 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Faster 2.04 Windows 11 X64 AMD Ryzen 9 5900X
Faster 2.01 Windows 11 X64 AMD Ryzen 9 5950X
Faster 2.37 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 1.84 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Faster 2.34 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Faster 2.72 ubuntu 18.04 X64 Intel Xeon CPU E5530 2.40GHz
Faster 2.04 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Faster 2.18 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Faster 2.25 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 2.13 macOS Big Sur 11.6.6 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

@dakersnar dakersnar mentioned this issue Jun 10, 2022
Closed
14 tasks
@dakersnar dakersnar removed the untriaged New issue has not been triaged by the area owner label Jul 15, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Aug 15, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime arch-x64 and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-System.Buffers runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jeffhandley @danmoseley @DrewScoggins @jeffschwMSFT @AndyAyersMS @kunalspathak @dakersnar