Regressions in Utf8/16 #77552
Description
Run Information
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
| Compare | ecf677d2a596b537a6b237e51a65c5a718a4dbc7 |
| Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Deep
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteDeepUtf8 - Duration of single invocation | 4.68 ms | 5.20 ms | 1.11 | 0.00 | False | |||||
| WriteDeepUtf16 - Duration of single invocation | 4.96 ms | 5.49 ms | 1.11 | 0.00 | False | |||||
| WriteDeepUtf16 - Duration of single invocation | 4.69 ms | 5.21 ms | 1.11 | 0.00 | False | |||||
| WriteDeepUtf8 - Duration of single invocation | 4.96 ms | 5.48 ms | 1.11 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Deep*'Related Issues
Regressions
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.200260992907801 > 4.910752926775149.
IsChangePoint: Marked as a change because one of 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -140.77110348536542 (T) = (0 -5181952.23370854) / Math.Sqrt((41577046.760174386 / (31)) + (80076987.6139092 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.10756411427746128 = (4678692.787991851 - 5181952.23370854) / 4678692.787991851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.486287175324676 > 5.212818396781789.
IsChangePoint: Marked as a change because one of 8/17/2022 3:53:31 PM, 9/28/2022 11:19:50 AM, 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -77.69372289541708 (T) = (0 -5499024.081632653) / Math.Sqrt((193279985.992411 / (31)) + (283192577.9323242 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.10685838048429365 = (4968137.007036633 - 5499024.081632653) / 4968137.007036633 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.206892907801419 > 4.918876000000001.
IsChangePoint: Marked as a change because one of 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -85.10006969424502 (T) = (0 -5201603.9736352805) / Math.Sqrt((71608249.70318332 / (31)) + (241317682.0209274 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.11015525674400864 = (4685474.3442742815 - 5201603.9736352805) / 4685474.3442742815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.482629090909091 > 5.209785285714286.
IsChangePoint: Marked as a change because one of 8/17/2022 2:33:27 AM, 9/28/2022 6:40:13 PM, 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -68.3446269302034 (T) = (0 -5485196.068851784) / Math.Sqrt((95523834.35597189 / (31)) + (385759911.8963819 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.10502919870109192 = (4963847.177340983 - 5485196.068851784) / 4963847.177340983 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
| Architecture | arm64 |
|---|---|
| OS | Windows 10.0.19041 |
| Baseline | 5108757b997c59ab8ba1fc5309ab0d4e730e2b77 |
| Compare | ecf677d2a596b537a6b237e51a65c5a718a4dbc7 |
| Diff | Diff |
Regressions in System.Text.Json.Tests.Perf_Basic
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| WriteBasicUtf16 - Duration of single invocation | 736.17 ns | 789.94 ns | 1.07 | 0.00 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 934.72 ns | 991.05 ns | 1.06 | 0.00 | False | |||||
| WriteBasicUtf16 - Duration of single invocation | 784.74 ns | 838.38 ns | 1.07 | 0.00 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Perf_Basic*'Related Issues
Regressions
Improvements
Payloads
Histogram
Edge Detector Info
System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 789.9433980213089 > 771.050289014384.
IsChangePoint: Marked as a change because one of 9/27/2022 6:39:02 PM, 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -86.1305970116965 (T) = (0 -788.6076260338984) / Math.Sqrt((3.2985038725694036 / (31)) + (1.9620629540210885 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.07286654608704811 = (735.0472702407425 - 788.6076260338984) / 735.0472702407425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 991.0493673521105 > 981.5575560083629.
IsChangePoint: Marked as a change because one of 10/10/2022 11:56:05 PM, 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -66.84780331816621 (T) = (0 -989.3013886383425) / Math.Sqrt((6.452867454118627 / (31)) + (2.867536274132608 / (7))) is less than -2.028094000977961 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (7) - 2, .025) and -0.05608996460338987 = (936.7586302269906 - 989.3013886383425) / 936.7586302269906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/10/2022 11:56:05 PM, 10/25/2022 8:54:59 AM, 10/27/2022 12:34:03 AM falls between 10/18/2022 9:26:37 AM and 10/27/2022 12:34:03 AM.
IsRegressionStdDev: Marked as regression because -54.41369476256814 (T) = (0 -842.0109717784655) / Math.Sqrt((1.8334500879113822 / (25)) + (7.179219278223537 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.07266795032924067 = (784.9688913703648 - 842.0109717784655) / 784.9688913703648 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository