Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 23 Regressions on 1/15/2023 10:11:50 PM #11812

Closed
performanceautofiler bot opened this issue Jan 17, 2023 · 3 comments
Closed

[Perf] Linux/arm64: 23 Regressions on 1/15/2023 10:11:50 PM #11812

performanceautofiler bot opened this issue Jan 17, 2023 · 3 comments
Assignees
@jandupej
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString4 - Duration of single invocation 282.20 ns 518.39 ns 1.84 0.07 False
TryFormat2 - Duration of single invocation 128.15 ns 252.25 ns 1.97 0.05 False
TryFormat4 - Duration of single invocation 229.06 ns 513.22 ns 2.24 0.06 False
TryFormat3 - Duration of single invocation 180.64 ns 361.73 ns 2.00 0.04 False
ToString3 - Duration of single invocation 227.05 ns 407.12 ns 1.79 0.03 False
ToString2 - Duration of single invocation 176.75 ns 289.99 ns 1.64 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.ToString4

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 518.3933850288806 > 297.2185183205381.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -31.30763292467035 (T) = (0 -520.8933178003447) / Math.Sqrt((1217.5510665655727 / (29)) + (82.65652153313515 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.907401229589154 = (273.0905850954824 - 520.8933178003447) / 273.0905850954824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormat2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 252.2509953335034 > 135.07487000789942.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -10.468324247959641 (T) = (0 -257.31727584175917) / Math.Sqrt((298.7768224429304 / (29)) + (614.778885563809 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.0876248471918089 = (123.25838916311629 - 257.31727584175917) / 123.25838916311629 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormat4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 513.219790831496 > 241.14954417960448.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -14.009259754068745 (T) = (0 -501.1741939085715) / Math.Sqrt((973.9835333790621 / (29)) + (1424.8290474701837 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.2315374926802913 = (224.58694758769795 - 501.1741939085715) / 224.58694758769795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormat3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 361.73277315660823 > 189.33212675061367.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -39.514174421716355 (T) = (0 -359.44849092888603) / Math.Sqrt((579.5809909434189 / (29)) + (7.042651568373354 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.0518493888610678 = (175.18268781336198 - 359.44849092888603) / 175.18268781336198 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 407.12352601611093 > 238.66953425363985.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -33.69567241469823 (T) = (0 -408.97539514945066) / Math.Sqrt((755.4698355244612 / (29)) + (22.261645836013344 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.8631226343091074 = (219.51072227787571 - 408.97539514945066) / 219.51072227787571 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 289.99491180503617 > 184.91569132922888.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -30.42392562429144 (T) = (0 -292.66179952256743) / Math.Sqrt((435.1413484292566 / (29)) + (4.893085927121546 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.7205124198447704 = (170.10153262885032 - 292.66179952256743) / 170.10153262885032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Byte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 61.10 ns 123.29 ns 2.02 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Byte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Byte.ToString(value: 255)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.29189239771264 > 64.08943196844815.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -12.919256180504641 (T) = (0 -130.48973787072285) / Math.Sqrt((51.31715727652316 / (29)) + (113.37483601218293 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.18962346060201 = (59.594601637509996 - 130.48973787072285) / 59.594601637509996 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 103.09 ns 175.49 ns 1.70 0.04 False
ToString - Duration of single invocation 83.19 ns 157.93 ns 1.90 0.25 False
TryFormat - Duration of single invocation 55.05 ns 122.55 ns 2.23 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryFormat(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 175.49398868868207 > 108.52347686059835.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -25.046853139391597 (T) = (0 -177.3844151109116) / Math.Sqrt((246.514720880968 / (29)) + (3.5385937867923922 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.7623382959235112 = (100.65287437787734 - 177.3844151109116) / 100.65287437787734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 157.93174328761 > 87.46145421410287.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -19.897300437887612 (T) = (0 -161.59292722523105) / Math.Sqrt((116.33757830089691 / (29)) + (48.92867765336518 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.9851802926418726 = (81.39962290789398 - 161.59292722523105) / 81.39962290789398 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.TryFormat(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 122.54674375250407 > 57.39506357543181.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -24.004226257792986 (T) = (0 -116.51698687027364) / Math.Sqrt((52.15388993229916 / (29)) + (20.4305723721296 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.1805208774501914 = (53.435391550354545 - 116.51698687027364) / 53.435391550354545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Perf_Convert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ChangeType - Duration of single invocation 113.49 ns 195.64 ns 1.72 0.22 False
ToDateTime_String - Duration of single invocation 544.26 ns 717.89 ns 1.32 0.16 False
ToDateTime_String - Duration of single invocation 842.93 ns 1.12 μs 1.33 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ChangeType

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.64150506727984 > 119.5331875627133.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -17.947982476390344 (T) = (0 -189.08759685957614) / Math.Sqrt((119.00255079428516 / (29)) + (57.06472433011126 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.6858330101927991 = (112.16270871214657 - 189.08759685957614) / 112.16270871214657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Perf_Convert.ToDateTime_String(value: "12/12/1999")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 717.8908912826823 > 556.0558242779923.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/11/2023 1:25:06 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -17.968433373438753 (T) = (0 -716.1471046737704) / Math.Sqrt((1786.6858267832438 / (22)) + (412.3800421909065 / (11))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (11) - 2, .025) and -0.3761998957982898 = (520.38014743371 - 716.1471046737704) / 520.38014743371 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Perf_Convert.ToDateTime_String(value: "February 26, 2009")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1226367729088553 > 874.674837711032.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/11/2023 1:25:06 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -17.6977521281601 (T) = (0 -1133.0227635976896) / Math.Sqrt((4894.042041295501 / (22)) + (723.1655286770713 / (11))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (11) - 2, .025) and -0.3608602757600211 = (832.5783210659981 - 1133.0227635976896) / 832.5783210659981 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 73.84 ns 136.01 ns 1.84 0.21 False
ToString - Duration of single invocation 73.45 ns 135.32 ns 1.84 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 136.01047015600153 > 77.29488017226666.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -29.237218231314106 (T) = (0 -136.01027204815608) / Math.Sqrt((87.90622485678898 / (29)) + (6.919557849457579 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.8834188611889292 = (72.21456408390117 - 136.01027204815608) / 72.21456408390117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt16.ToString(value: 65535)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.31903371290426 > 77.27159624296235.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -14.734695543466803 (T) = (0 -140.15716001431252) / Math.Sqrt((87.14680268503432 / (29)) + (72.95793996859366 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.9400986554427714 = (72.24228501015362 - 140.15716001431252) / 72.24228501015362 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 75.85 ns 135.26 ns 1.78 0.16 False
TryFormat - Duration of single invocation 48.34 ns 106.65 ns 2.21 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.2569006320894 > 79.73832039637735.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -24.595919723298394 (T) = (0 -137.91664538204287) / Math.Sqrt((91.74745986731341 / (29)) + (13.983336515355289 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.8526175214839936 = (74.44420868456871 - 137.91664538204287) / 74.44420868456871 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.64705343496699 > 51.05132438560979.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -38.238221719326475 (T) = (0 -104.83397295795035) / Math.Sqrt((39.595032519059146 / (29)) + (3.4533807227677134 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.1955268674508328 = (47.748890943735184 - 104.83397295795035) / 47.748890943735184 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 63.87 ns 146.35 ns 2.29 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.ToString(value: 127)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 146.34702744099272 > 66.93208171110663.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -12.865050120027567 (T) = (0 -131.03190123032977) / Math.Sqrt((54.47790172985848 / (29)) + (107.77781818143659 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.114671866612271 = (61.96323093863465 - 131.03190123032977) / 61.96323093863465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 75.28 ns 156.55 ns 2.08 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.ToString(value: 32767)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 156.5548803884584 > 79.46221361567447.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -15.136526788087266 (T) = (0 -143.88049915797347) / Math.Sqrt((89.3118255616012 / (29)) + (72.53030703953962 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.939966418381 = (74.16648958183978 - 143.88049915797347) / 74.16648958183978 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 73.86 ns 137.06 ns 1.86 0.23 False
TryFormat - Duration of single invocation 48.92 ns 117.37 ns 2.40 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.0567077387886 > 77.51722252529997.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -34.93711695576984 (T) = (0 -136.80265260325876) / Math.Sqrt((86.76198265790568 / (29)) + (1.6181806006377688 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.8891089461012717 = (72.41649714569984 - 136.80265260325876) / 72.41649714569984 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 117.3715162977933 > 49.71181089145755.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -11.361244587287167 (T) = (0 -112.8401124396114) / Math.Sqrt((41.371395527954405 / (29)) + (132.64104579682288 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -1.4517690524340259 = (46.02395659068618 - 112.8401124396114) / 46.02395659068618 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 2eca2d319bf8d1247edf38facfdd501e0e2f0030
Compare 51a8dd5323721b363e61069575511f783e7ea6d3
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 85.72 ns 168.30 ns 1.96 0.23 False
TryFormat - Duration of single invocation 106.09 ns 177.34 ns 1.67 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 168.30295808392117 > 89.2528362514212.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -26.501035467390782 (T) = (0 -162.22356212970993) / Math.Sqrt((124.15594926072579 / (29)) + (18.33136625480424 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.9469225030663942 = (83.32307108999385 - 162.22356212970993) / 83.32307108999385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 177.33673805974175 > 110.41773800110636.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 1/15/2023 4:47:52 PM, 1/17/2023 2:54:07 AM falls between 1/8/2023 8:00:22 AM and 1/17/2023 2:54:07 AM.
IsRegressionStdDev: Marked as regression because -16.32163720261547 (T) = (0 -180.55125301073167) / Math.Sqrt((248.81873526779827 / (29)) + (59.83029366050442 / (4))) is less than -2.0395134463927462 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (29) + (4) - 2, .025) and -0.7811811182758703 = (101.36602682241536 - 180.55125301073167) / 101.36602682241536 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@lewing
Copy link
Member

lewing commented Jan 29, 2023

dotnet/runtime#80745

@EgorBo
Copy link
Member

EgorBo commented Jan 31, 2023

dotnet/runtime#80762

@naricc naricc assigned jandupej and unassigned EgorBo Jan 31, 2023
@naricc naricc added LinkedFromMaster Is this issue linked from the master perf issue and removed untriaged labels Jan 31, 2023
@naricc naricc mentioned this issue Jan 31, 2023
Open
@kotlarmilos
Copy link
Member

Mitigated by dotnet/runtime#81695.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jandupej jandupej

Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lewing @EgorBo @kotlarmilos @naricc @jandupej