Skip to content

[Perf] Windows/x64: 7 Regressions on 10/25/2022 12:27:43 PM #9428

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 891.21 ns 949.21 ns 1.07 0.04 False
WriteBasicUtf16 - Duration of single invocation 848.10 ns 915.21 ns 1.08 0.05 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 949.2147514014134 > 940.0068051150469.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -22.422580298695127 (T) = (0 -951.7096492354435) / Math.Sqrt((55.66860440376942 / (19)) + (111.78511781004951 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.06294473658201595 = (895.3519562039907 - 951.7096492354435) / 895.3519562039907 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 915.2144381379795 > 891.2485808182577.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -7.994657054621247 (T) = (0 -909.4659434069454) / Math.Sqrt((608.1230664348213 / (19)) + (387.0310791058124 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.06172158638025719 = (856.5955096642622 - 909.4659434069454) / 856.5955096642622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS Windows 10.0.19042
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Strings

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteStringsUtf16 - Duration of single invocation 6.82 ms 7.34 ms 1.08 0.16 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Strings*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Strings.WriteStringsUtf16(Formatted: False, SkipValidation: False, Escaped: NoneEscaped)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.3350139393939395 > 7.119729858585859.
IsChangePoint: Marked as a change because one of 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -22.63418842249305 (T) = (0 -7388989.713963993) / Math.Sqrt((11732914861.085526 / (19)) + (7376423522.784775 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.09749769364944429 = (6732578.80788234 - 7388989.713963993) / 6732578.80788234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline 5108757b997c59ab8ba1fc5309ab0d4e730e2b77
Compare f4b715f1cfadcbf85486f369724528c051be5733
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf8 - Duration of single invocation 4.26 ms 4.69 ms 1.10 0.14 False
WriteDeepUtf8 - Duration of single invocation 4.49 ms 4.88 ms 1.09 0.41 False
WriteDeepUtf16 - Duration of single invocation 4.20 ms 4.67 ms 1.11 0.02 False
WriteDeepUtf16 - Duration of single invocation 4.33 ms 4.93 ms 1.14 0.37 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.694830062893082 > 4.383590147540984.
IsChangePoint: Marked as a change because one of 9/12/2022 12:30:55 AM, 9/12/2022 4:32:08 PM, 9/29/2022 7:42:19 AM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -16.88691530744569 (T) = (0 -4746526.217096403) / Math.Sqrt((4578301164.950006 / (19)) + (33649764028.719597 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.1445838718771823 = (4146944.8711625054 - 4746526.217096403) / 4146944.8711625054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.880972829131653 > 4.595497236842106.
IsChangePoint: Marked as a change because one of 9/8/2022 11:29:08 PM, 10/10/2022 9:53:50 PM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -12.42650000831821 (T) = (0 -5407388.196774259) / Math.Sqrt((64261975070.34909 / (19)) + (86924151113.33223 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.21691241860997265 = (4443531.115370561 - 5407388.196774259) / 4443531.115370561 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.673801257861634 > 4.4003929685870276.
IsChangePoint: Marked as a change because one of 8/24/2022 2:39:26 AM, 9/3/2022 11:44:47 AM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -42.28656358859927 (T) = (0 -4678072.509339407) / Math.Sqrt((2379087833.5240636 / (19)) + (737104976.2759154 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.1233447151345498 = (4164414.045228393 - 4678072.509339407) / 4164414.045228393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.9314011764705885 > 4.537332057017544.
IsChangePoint: Marked as a change because one of 9/9/2022 3:12:19 AM, 9/13/2022 12:23:36 PM, 9/29/2022 7:42:19 AM, 10/25/2022 8:17:52 AM, 11/1/2022 10:15:31 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 10:15:31 AM.
IsRegressionStdDev: Marked as regression because -17.988394779724985 (T) = (0 -5119272.103058663) / Math.Sqrt((890545861.9605262 / (19)) + (60129826569.67633 / (33))) is less than -2.0085591120996775 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (33) - 2, .025) and -0.17912060179071096 = (4341601.779567 - 5119272.103058663) / 4341601.779567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions