Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x64: 4 Regressions from "Add binary parsing support to integer types" #85340

Closed
performanceautofiler bot opened this issue Apr 25, 2023 · 8 comments
Closed

[Perf] Windows/x64: 4 Regressions from "Add binary parsing support to integer types" #85340

performanceautofiler bot opened this issue Apr 25, 2023 · 8 comments
Labels
arch-x64 area-System.Text.Json os-windows runtime-coreclr specific to the CoreCLR runtime

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline f20822f9f0e0d8d2faa40f41b8e04d651c5bf753
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 318.92 ns 338.77 ns 1.06 0.07 False
SerializeToWriter - Duration of single invocation 216.81 ns 232.12 ns 1.07 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 338.765100049601 > 334.86168994967016.
IsChangePoint: Marked as a change because one of 4/19/2023 4:18:46 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -6.622461662382049 (T) = (0 -348.61663890200265) / Math.Sqrt((225.21843564438967 / (23)) + (28.18455149580087 / (26))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (26) - 2, .025) and -0.06683529249626141 = (326.77643995661526 - 348.61663890200265) / 326.77643995661526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.12006739049045 > 221.76749914670623.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -22.100545768697337 (T) = (0 -235.97498687774075) / Math.Sqrt((6.319398167060556 / (12)) + (22.595670474912442 / (35))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (35) - 2, .025) and -0.1128425594302943 = (212.04705452543544 - 235.97498687774075) / 212.04705452543544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteByteArrayAsBase64_HeavyEscaping - Duration of single invocation 65.47 ns 79.97 ns 1.22 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.96827338309903 > 69.23090764851216.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 2/24/2023 9:01:57 AM, 3/15/2023 5:33:41 AM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.649543007114763 (T) = (0 -73.45448822507512) / Math.Sqrt((1.7207269844399058 / (22)) + (5.882775793110427 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.12365602433707672 = (65.37097353116677 - 73.45448822507512) / 65.37097353116677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 13.84 ns 15.88 ns 1.15 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.875352741167239 > 14.672938260882662.
IsChangePoint: Marked as a change because one of 3/6/2023 8:17:12 PM, 3/31/2023 8:05:26 PM, 4/4/2023 4:23:54 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.254305319500027 (T) = (0 -15.46321322725647) / Math.Sqrt((0.0907390688218966 / (22)) + (0.06243086532401762 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.08659084109612292 = (14.230943831311523 - 15.46321322725647) / 14.230943831311523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-windows runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Apr 25, 2023
@kunalspathak
Copy link
Member

#84998

@kunalspathak kunalspathak changed the title [Perf] Windows/x64: 4 Regressions on 4/19/2023 10:34:24 AM [Perf] Windows/x64: 4 Regressions from "Add binary parsing support to integer types" Apr 25, 2023
@kunalspathak kunalspathak removed their assignment Apr 25, 2023
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Apr 25, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Apr 25, 2023
@kunalspathak
Copy link
Member

@stephentoub

@stephentoub
Copy link
Member

I don't think the JSON tests are related to #84998.

The ParseSpan one could be, but it's also a 2ns change on a test that's shown larger spikes in the past. The change could potentially have impacted inlining somewhere maybe, but the bulk of the changes should be on code paths that aren't exercised by this particular test, and those that are should have been constant folded / dead code eliminated by the JIT. I'll let this sit for a bit.

@ghost
Copy link

ghost commented Apr 26, 2023

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline f20822f9f0e0d8d2faa40f41b8e04d651c5bf753
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 318.92 ns 338.77 ns 1.06 0.07 False
SerializeToWriter - Duration of single invocation 216.81 ns 232.12 ns 1.07 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 338.765100049601 > 334.86168994967016.
IsChangePoint: Marked as a change because one of 4/19/2023 4:18:46 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -6.622461662382049 (T) = (0 -348.61663890200265) / Math.Sqrt((225.21843564438967 / (23)) + (28.18455149580087 / (26))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (26) - 2, .025) and -0.06683529249626141 = (326.77643995661526 - 348.61663890200265) / 326.77643995661526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.12006739049045 > 221.76749914670623.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -22.100545768697337 (T) = (0 -235.97498687774075) / Math.Sqrt((6.319398167060556 / (12)) + (22.595670474912442 / (35))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (35) - 2, .025) and -0.1128425594302943 = (212.04705452543544 - 235.97498687774075) / 212.04705452543544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteByteArrayAsBase64_HeavyEscaping - Duration of single invocation 65.47 ns 79.97 ns 1.22 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.96827338309903 > 69.23090764851216.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 2/24/2023 9:01:57 AM, 3/15/2023 5:33:41 AM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.649543007114763 (T) = (0 -73.45448822507512) / Math.Sqrt((1.7207269844399058 / (22)) + (5.882775793110427 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.12365602433707672 = (65.37097353116677 - 73.45448822507512) / 65.37097353116677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 13.84 ns 15.88 ns 1.15 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.875352741167239 > 14.672938260882662.
IsChangePoint: Marked as a change because one of 3/6/2023 8:17:12 PM, 3/31/2023 8:05:26 PM, 4/4/2023 4:23:54 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.254305319500027 (T) = (0 -15.46321322725647) / Math.Sqrt((0.0907390688218966 / (22)) + (0.06243086532401762 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.08659084109612292 = (14.230943831311523 - 15.46321322725647) / 14.230943831311523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Globalization, os-windows, arch-x64, untriaged, runtime-coreclr, needs-area-label
Milestone: -

@tarekgh tarekgh added area-Meta and removed area-System.Globalization needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Apr 26, 2023
@ghost
Copy link

ghost commented Apr 26, 2023

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline f20822f9f0e0d8d2faa40f41b8e04d651c5bf753
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 318.92 ns 338.77 ns 1.06 0.07 False
SerializeToWriter - Duration of single invocation 216.81 ns 232.12 ns 1.07 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 338.765100049601 > 334.86168994967016.
IsChangePoint: Marked as a change because one of 4/19/2023 4:18:46 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -6.622461662382049 (T) = (0 -348.61663890200265) / Math.Sqrt((225.21843564438967 / (23)) + (28.18455149580087 / (26))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (26) - 2, .025) and -0.06683529249626141 = (326.77643995661526 - 348.61663890200265) / 326.77643995661526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.12006739049045 > 221.76749914670623.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -22.100545768697337 (T) = (0 -235.97498687774075) / Math.Sqrt((6.319398167060556 / (12)) + (22.595670474912442 / (35))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (35) - 2, .025) and -0.1128425594302943 = (212.04705452543544 - 235.97498687774075) / 212.04705452543544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteByteArrayAsBase64_HeavyEscaping - Duration of single invocation 65.47 ns 79.97 ns 1.22 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.96827338309903 > 69.23090764851216.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 2/24/2023 9:01:57 AM, 3/15/2023 5:33:41 AM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.649543007114763 (T) = (0 -73.45448822507512) / Math.Sqrt((1.7207269844399058 / (22)) + (5.882775793110427 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.12365602433707672 = (65.37097353116677 - 73.45448822507512) / 65.37097353116677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 13.84 ns 15.88 ns 1.15 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.875352741167239 > 14.672938260882662.
IsChangePoint: Marked as a change because one of 3/6/2023 8:17:12 PM, 3/31/2023 8:05:26 PM, 4/4/2023 4:23:54 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.254305319500027 (T) = (0 -15.46321322725647) / Math.Sqrt((0.0907390688218966 / (22)) + (0.06243086532401762 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.08659084109612292 = (14.230943831311523 - 15.46321322725647) / 14.230943831311523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-Meta, os-windows, arch-x64, untriaged, runtime-coreclr
Milestone: -

@ghost
Copy link

ghost commented May 17, 2023

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline f20822f9f0e0d8d2faa40f41b8e04d651c5bf753
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 318.92 ns 338.77 ns 1.06 0.07 False
SerializeToWriter - Duration of single invocation 216.81 ns 232.12 ns 1.07 0.13 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 338.765100049601 > 334.86168994967016.
IsChangePoint: Marked as a change because one of 4/19/2023 4:18:46 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -6.622461662382049 (T) = (0 -348.61663890200265) / Math.Sqrt((225.21843564438967 / (23)) + (28.18455149580087 / (26))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (26) - 2, .025) and -0.06683529249626141 = (326.77643995661526 - 348.61663890200265) / 326.77643995661526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.12006739049045 > 221.76749914670623.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -22.100545768697337 (T) = (0 -235.97498687774075) / Math.Sqrt((6.319398167060556 / (12)) + (22.595670474912442 / (35))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (12) + (35) - 2, .025) and -0.1128425594302943 = (212.04705452543544 - 235.97498687774075) / 212.04705452543544 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteByteArrayAsBase64_HeavyEscaping - Duration of single invocation 65.47 ns 79.97 ns 1.22 0.07 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Base64*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Base64.WriteByteArrayAsBase64_HeavyEscaping(NumberOfBytes: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 79.96827338309903 > 69.23090764851216.
IsChangePoint: Marked as a change because one of 2/16/2023 4:50:02 PM, 2/24/2023 9:01:57 AM, 3/15/2023 5:33:41 AM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -14.649543007114763 (T) = (0 -73.45448822507512) / Math.Sqrt((1.7207269844399058 / (22)) + (5.882775793110427 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.12365602433707672 = (65.37097353116677 - 73.45448822507512) / 65.37097353116677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.19042
Queue OwlWindows
Baseline 9cc81637a1fffbda47ed7d42a8372b5c88100438
Compare 33b18692fdf2ef438de3da38bc419c6616a746c5
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseSpan - Duration of single invocation 13.84 ns 15.88 ns 1.15 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ParseSpan(value: "12345")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.875352741167239 > 14.672938260882662.
IsChangePoint: Marked as a change because one of 3/6/2023 8:17:12 PM, 3/31/2023 8:05:26 PM, 4/4/2023 4:23:54 PM, 4/19/2023 1:39:34 AM, 4/25/2023 3:46:39 AM falls between 4/12/2023 8:39:35 PM and 4/25/2023 3:46:39 AM.
IsRegressionStdDev: Marked as regression because -15.254305319500027 (T) = (0 -15.46321322725647) / Math.Sqrt((0.0907390688218966 / (22)) + (0.06243086532401762 / (26))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (26) - 2, .025) and -0.08659084109612292 = (14.230943831311523 - 15.46321322725647) / 14.230943831311523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-System.Text.Json, os-windows, arch-x64, untriaged, runtime-coreclr
Milestone: -

@ericstj
Copy link
Member

ericstj commented May 17, 2023

@stephentoub mentioned that this JSON test regression didn't seem to be caused by above mentioned PR. Might make sense to look at the latest data and see if a JSON change (or something else) could have caused this.

@stephentoub
Copy link
Member

All of the benchmarks here either returned to where they were or were just experiencing historical bimodal behavior.

@stephentoub stephentoub closed this as not planned Won't fix, can't repro, duplicate, stale May 17, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label May 17, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Jun 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-System.Text.Json os-windows runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@stephentoub @ericstj @jeffschwMSFT @tarekgh @kunalspathak