Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 398 Improvements on 4/4/2023 3:37:20 PM #15797

Closed
performanceautofiler bot opened this issue Apr 11, 2023 · 24 comments
Closed

[Perf] Linux/arm64: 398 Improvements on 4/4/2023 3:37:20 PM #15797

performanceautofiler bot opened this issue Apr 11, 2023 · 24 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-improvement runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.19 ns 1.00 ns 0.08 0.14 False
AddOperatorBenchmark - Duration of single invocation 36.03 ns 1.01 ns 0.03 0.11 False
UnaryNegateOperatorBenchmark - Duration of single invocation 32.81 ns 1.13 ns 0.03 0.08 False
AddBenchmark - Duration of single invocation 36.57 ns 1.06 ns 0.03 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 35.05 ns 1.07 ns 0.03 0.10 False
AllBitsSetBenchmark - Duration of single invocation 6.50 ns 1.16 ns 0.18 0.18 True
MultiplyBenchmark - Duration of single invocation 36.23 ns 1.00 ns 0.03 0.04 True
XorBenchmark - Duration of single invocation 13.18 ns 1.04 ns 0.08 0.13 False
SubtractBenchmark - Duration of single invocation 35.98 ns 1.01 ns 0.03 0.10 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.19 ns 1.15 ns 0.09 0.12 False
BitwiseOrBenchmark - Duration of single invocation 13.18 ns 1.03 ns 0.08 0.12 False
OnesComplementBenchmark - Duration of single invocation 11.13 ns 1.62 ns 0.15 0.15 False
MultiplyOperatorBenchmark - Duration of single invocation 36.11 ns 1.17 ns 0.03 0.05 True
EqualityOperatorBenchmark - Duration of single invocation 3.29 ns 0.02 ns 0.01 0.27 False
NegateBenchmark - Duration of single invocation 32.64 ns 0.99 ns 0.03 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.12 ns 1.01 ns 0.09 0.16 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.30 ns 1.01 ns 0.08 0.12 False
BitwiseAndBenchmark - Duration of single invocation 13.25 ns 1.05 ns 0.08 0.12 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0022624489707959 < 12.701813462443234.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.840299985435685 (T) = (0 -1.1844835047783475) / Math.Sqrt((16.292517613443774 / (299)) + (0.07454169342601547 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9065805905910096 = (12.679201380868038 - 1.1844835047783475) / 12.679201380868038 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0060193735811305 < 34.10312985536552.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.59330650728501 (T) = (0 -1.1865091649494572) / Math.Sqrt((116.46976721118538 / (299)) + (0.06924976234380173 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9632184651620007 = (32.25828313514708 - 1.1865091649494572) / 32.25828313514708 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1318773785890635 < 31.027625902435197.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.45156614555222 (T) = (0 -1.149598146611813) / Math.Sqrt((97.7395303178968 / (299)) + (0.008408344136209212 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9609496964082649 = (29.43890420496301 - 1.149598146611813) / 29.43890420496301 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0557354548863345 < 34.22802700074242.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.57750035706735 (T) = (0 -1.1284530947443527) / Math.Sqrt((128.06427283963038 / (299)) + (0.007946498438696124 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9664027635865112 = (33.5876761069341 - 1.1284530947443527) / 33.5876761069341 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0732849338760786 < 33.69614994489321.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.85360277955107 (T) = (0 -1.1455377816359753) / Math.Sqrt((116.46736789640333 / (299)) + (0.012156539855889823 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9645137342771476 = (32.28115887376321 - 1.1455377816359753) / 32.28115887376321 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1633291000913277 < 6.292653858773745.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 169.70192943109006 (T) = (0 -1.140134740094411) / Math.Sqrt((0.0770752552235925 / (299)) + (0.01266486719621311 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.811215900635641 = (6.039357890485874 - 1.140134740094411) / 6.039357890485874 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9967726833054023 < 34.41745566387768.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 416.1457305431207 (T) = (0 -1.1109224090910965) / Math.Sqrt((2.2596444891166274 / (299)) + (0.004028054835899834 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9705360630538044 = (37.70447958532359 - 1.1109224090910965) / 37.70447958532359 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.042123350821531 < 12.710590940384959.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.37280885000038 (T) = (0 -1.1804240263027637) / Math.Sqrt((18.990382369993974 / (299)) + (0.017583370653374363 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9121994708792142 = (13.44438397038442 - 1.1804240263027637) / 13.44438397038442 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.014319243403874 < 34.186234837672494.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.600593211119055 (T) = (0 -1.139294831972795) / Math.Sqrt((128.08340978960305 / (299)) + (0.018976539384028806 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9661269145719281 = (33.634220726424374 - 1.139294831972795) / 33.634220726424374 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1472266146266605 < 12.665446327046501.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.8613955177741 (T) = (0 -1.1159863723701615) / Math.Sqrt((15.993768645238811 / (299)) + (0.00537311477320508 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9119407505276548 = (12.673130637124428 - 1.1159863723701615) / 12.673130637124428 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0311581865552344 < 12.72488596932493.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.60689258125941 (T) = (0 -1.173591579298094) / Math.Sqrt((19.031355023954745 / (299)) + (0.012137222085235239 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9129856410430816 = (13.487332359468963 - 1.173591579298094) / 13.487332359468963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.622907479291253 < 10.811987157318452.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.09016561644735 (T) = (0 -1.215328254649883) / Math.Sqrt((7.355882901648393 / (299)) + (0.0681489179336521 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8576138596871978 = (8.535439277867768 - 1.215328254649883) / 8.535439277867768 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.16640422100869 < 34.299189921003844.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 835.7477758560349 (T) = (0 -1.125270698581238) / Math.Sqrt((0.4305552495600273 / (299)) + (0.007745578915115041 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9690462467363753 = (36.35328772564724 - 1.125270698581238) / 36.35328772564724 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.017534892034348215 < 2.9792918495512533.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 54.948744446947146 (T) = (0 -0.08924016147737894) / Math.Sqrt((2.1267019562167984 / (299)) + (0.05851412603435975 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9837420047931009 = (5.489001586094096 - 0.08924016147737894) / 5.489001586094096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9940815687466537 < 31.12579567482411.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.48638523622576 (T) = (0 -1.1392221558608038) / Math.Sqrt((97.79690315079178 / (299)) + (0.00815451452957481 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9613249661868435 = (29.456267869461108 - 1.1392221558608038) / 29.456267869461108 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0102889019919383 < 10.632081149810517.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.18995585437535 (T) = (0 -1.166065404781679) / Math.Sqrt((7.326714811197396 / (299)) + (0.01531041012452959 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8627203666082242 = (8.494088860609791 - 1.166065404781679) / 8.494088860609791 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0099358978509103 < 12.641777804482743.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.50390097218929 (T) = (0 -1.1227012577776767) / Math.Sqrt((16.15074330053114 / (299)) + (0.00998030306591847 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9114328419384263 = (12.67627055389032 - 1.1227012577776767) / 12.67627055389032 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0469125428280142 < 12.736689821882106.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.437263164606655 (T) = (0 -1.1806984451317704) / Math.Sqrt((19.101980341704333 / (299)) + (0.012052966132562697 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9123699925119424 = (13.473677327856885 - 1.1806984451317704) / 13.473677327856885 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 13.31 ns 1.00 ns 0.08 0.12 False
AddBenchmark - Duration of single invocation 15.58 ns 1.01 ns 0.06 0.12 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.11 ns 1.09 ns 0.10 0.17 False
NegateBenchmark - Duration of single invocation 14.68 ns 1.15 ns 0.08 0.11 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.28 ns 1.02 ns 0.08 0.13 False
OnesComplementBenchmark - Duration of single invocation 11.14 ns 1.10 ns 0.10 0.17 False
XorBenchmark - Duration of single invocation 13.20 ns 1.05 ns 0.08 0.13 False
AddOperatorBenchmark - Duration of single invocation 15.57 ns 0.99 ns 0.06 0.15 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.15 ns 1.25 ns 0.09 0.14 False
AllBitsSetBenchmark - Duration of single invocation 6.49 ns 1.26 ns 0.19 0.16 True
EqualityOperatorBenchmark - Duration of single invocation 3.03 ns 0.03 ns 0.01 0.43 False
UnaryNegateOperatorBenchmark - Duration of single invocation 14.39 ns 1.08 ns 0.07 0.13 False
MultiplyBenchmark - Duration of single invocation 15.75 ns 3.19 ns 0.20 0.09 True
SubtractBenchmark - Duration of single invocation 15.67 ns 1.03 ns 0.07 0.13 False
MultiplyOperatorBenchmark - Duration of single invocation 15.69 ns 0.92 ns 0.06 0.05 True
BitwiseOrBenchmark - Duration of single invocation 13.10 ns 1.05 ns 0.08 0.12 False
SubtractionOperatorBenchmark - Duration of single invocation 15.67 ns 0.96 ns 0.06 0.12 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.20 ns 1.00 ns 0.08 0.14 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9993321697891718 < 12.734299246676402.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.12708949234093 (T) = (0 -1.1970812818880285) / Math.Sqrt((19.209245465522393 / (299)) + (0.07822716787679257 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9111186367848185 = (13.468304699489122 - 1.1970812818880285) / 13.468304699489122 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.01175853000589 < 15.025499154795007.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.88190697092742 (T) = (0 -1.1472845962621234) / Math.Sqrt((33.70605312224219 / (299)) + (0.012664607484078267 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.933540589837232 = (17.262936782801265 - 1.1472845962621234) / 17.262936782801265 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0920380843451938 < 10.69507969409962.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.49983180243531 (T) = (0 -1.1742140317996748) / Math.Sqrt((7.296417509608775 / (299)) + (0.016008906401059193 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8625948684420083 = (8.545634493309292 - 1.1742140317996748) / 8.545634493309292 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1521036986029114 < 13.970106540281662.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.9513358320361 (T) = (0 -1.149943561436605) / Math.Sqrt((18.957788859754267 / (299)) + (0.008638701776184447 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9148972667464712 = (13.512416317003806 - 1.149943561436605) / 13.512416317003806 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0190826085566331 < 12.697326235979592.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.52871935030455 (T) = (0 -1.1427982471704343) / Math.Sqrt((16.09088898896759 / (299)) + (0.011873241130417916 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9099275842157871 = (12.687549647921562 - 1.1427982471704343) / 12.687549647921562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1002342344225773 < 10.795970726213964.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.77164568436881 (T) = (0 -1.1450862532849224) / Math.Sqrt((7.350821760517005 / (299)) + (0.009770812712123406 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8659870426959615 = (8.544593570060824 - 1.1450862532849224) / 8.544593570060824 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0521621671103996 < 12.735492522555498.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.854981926548426 (T) = (0 -1.1035880011236547) / Math.Sqrt((19.109411142810185 / (299)) + (0.006004512195044406 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9181292660842041 = (13.479639772848433 - 1.1035880011236547) / 13.479639772848433 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9921779594714917 < 15.028637986239962.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.30976858983691 (T) = (0 -1.2071276836912723) / Math.Sqrt((27.018823355705546 / (299)) + (0.08245828239872866 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9246233556998409 = (16.014611620070813 - 1.2071276836912723) / 16.014611620070813 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2478190031411343 < 12.638921008847655.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.464758390150585 (T) = (0 -1.1847009817161578) / Math.Sqrt((16.067181163074853 / (299)) + (0.00868530019159381 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9066532492775226 = (12.691400317064145 - 1.1847009817161578) / 12.691400317064145 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2646000897946947 < 6.418441200189941.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 170.01358995680894 (T) = (0 -1.143389582906034) / Math.Sqrt((0.06233955733322748 / (299)) + (0.014526533979398874 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8116665654124023 = (6.071091866453595 - 1.143389582906034) / 6.071091866453595 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.026841566919768817 < 2.890304417941816.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 60.57155476788081 (T) = (0 -0.06435889683434044) / Math.Sqrt((2.07296212628873 / (299)) + (0.00868717862522339 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9877256890744365 = (5.243381663104314 - 0.06435889683434044) / 5.243381663104314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.07562360757354 < 13.927157839570755.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.92105875114655 (T) = (0 -1.1666783915988974) / Math.Sqrt((18.985215833910825 / (299)) + (0.011764462252234577 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9138571575721627 = (13.543532564254955 - 1.1666783915988974) / 13.543532564254955 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.18768927655051 < 15.134202387519693.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 124.3203716814545 (T) = (0 -1.2097421622656788) / Math.Sqrt((3.734803328889339 / (299)) + (0.19257496330487048 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9368839851715809 = (19.16696048624688 - 1.2097421622656788) / 19.16696048624688 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0316877762762517 < 15.18654232131746.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.756085796875645 (T) = (0 -1.1633273625654084) / Math.Sqrt((33.93420695512169 / (299)) + (0.014052112153477188 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9327359738906451 = (17.29494099378055 - 1.1633273625654084) / 17.29494099378055 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9246677665453518 < 15.121410756835875.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 154.04833358359627 (T) = (0 -1.2131517913063727) / Math.Sqrt((1.1907876488321278 / (299)) + (0.17956097733957666 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9323760794611503 = (17.939684384454182 - 1.2131517913063727) / 17.939684384454182 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.050989957937357 < 12.752066244252271.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.91652660519649 (T) = (0 -1.1099080706036584) / Math.Sqrt((19.00829075991871 / (299)) + (0.008890592962720777 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9176686888917466 = (13.480995937794493 - 1.1099080706036584) / 13.480995937794493 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9610116897009814 < 14.973596028001474.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.08814657995848 (T) = (0 -1.1238443974899937) / Math.Sqrt((27.303945932925853 / (299)) + (0.011971036121475993 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9297593025431743 = (15.999903733597996 - 1.1238443974899937) / 15.999903733597996 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.996437953806821 < 12.775630505911163.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.846570383474536 (T) = (0 -1.130713906140512) / Math.Sqrt((15.900713034391009 / (299)) + (0.009331325524349423 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9107527397476556 = (12.669452294036216 - 1.130713906140512) / 12.669452294036216 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractBenchmark - Duration of single invocation 14.10 ns 1.04 ns 0.07 0.11 False
AllBitsSetBenchmark - Duration of single invocation 6.58 ns 1.18 ns 0.18 0.17 True
SubtractionOperatorBenchmark - Duration of single invocation 13.70 ns 1.16 ns 0.08 0.11 False
BitwiseAndBenchmark - Duration of single invocation 13.24 ns 1.33 ns 0.10 0.12 False
OnesComplementBenchmark - Duration of single invocation 11.13 ns 1.06 ns 0.10 0.16 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.22 ns 1.10 ns 0.08 0.12 False
BitwiseAndOperatorBenchmark - Duration of single invocation 15.47 ns 1.31 ns 0.08 0.14 False
AddBenchmark - Duration of single invocation 13.78 ns 1.21 ns 0.09 0.11 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.16 ns 1.02 ns 0.09 0.16 False
AddOperatorBenchmark - Duration of single invocation 13.83 ns 1.26 ns 0.09 0.13 False
BitwiseOrBenchmark - Duration of single invocation 13.09 ns 1.26 ns 0.10 0.13 False
InequalityOperatorBenchmark - Duration of single invocation 3.24 ns 0.00 ns 0.00 0.38 False
NegateBenchmark - Duration of single invocation 12.57 ns 1.01 ns 0.08 0.13 False
EqualityOperatorBenchmark - Duration of single invocation 3.01 ns 0.24 ns 0.08 0.41 False
XorBenchmark - Duration of single invocation 13.12 ns 1.28 ns 0.10 0.11 False
UnaryNegateOperatorBenchmark - Duration of single invocation 12.51 ns 1.14 ns 0.09 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.18 ns 1.34 ns 0.10 0.13 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0426979544151795 < 13.194701985836678.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.65686518948601 (T) = (0 -1.2465395615004753) / Math.Sqrt((25.478043264926555 / (299)) + (0.03882633490960453 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9168992633779195 = (15.000343103688687 - 1.2465395615004753) / 15.000343103688687 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1794092599863826 < 6.350584648534857.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 139.63197529971896 (T) = (0 -1.183275438520373) / Math.Sqrt((0.07151031241512737 / (299)) + (0.02243161125552628 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8043949510324907 = (6.049309282997696 - 1.183275438520373) / 6.049309282997696 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1623854856396547 < 13.156538619892638.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.073318433563784 (T) = (0 -1.1707162461832612) / Math.Sqrt((19.49997708810553 / (299)) + (0.011726633702597317 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9148681151164921 = (13.751795203233625 - 1.1707162461832612) / 13.751795203233625 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.327937869362846 < 12.717232535014835.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.503281328560035 (T) = (0 -1.2894830987958514) / Math.Sqrt((19.069470434992805 / (299)) + (0.1154008993393429 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9043887469829852 = (13.486729418412477 - 1.2894830987958514) / 13.486729418412477 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.058389107314445 < 10.689630361950329.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.18591936008059 (T) = (0 -1.1583731659773069) / Math.Sqrt((7.475637567716191 / (299)) + (0.011472594787516372 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8642607124973124 = (8.53380909307021 - 1.1583731659773069) / 8.53380909307021 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.100342455822463 < 12.604369797400786.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.32626304055191 (T) = (0 -1.2945192252951403) / Math.Sqrt((16.030985802206434 / (299)) + (0.09696643738082732 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8978731547509332 = (12.67560181789684 - 1.2945192252951403) / 12.67560181789684 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3073820150401652 < 12.727215962330684.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.88916323369804 (T) = (0 -1.172226405742053) / Math.Sqrt((16.203896345282704 / (299)) + (0.028517047571524377 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9075738813273214 = (12.682847906806744 - 1.172226405742053) / 12.682847906806744 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2139295499745428 < 13.151701109349242.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.66480956610026 (T) = (0 -1.1378356286363605) / Math.Sqrt((25.371670562040634 / (299)) + (0.01143576995100295 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9244623774832746 = (15.063164430207252 - 1.1378356286363605) / 15.063164430207252 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0192760036717974 < 10.756394019719682.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.480644513106085 (T) = (0 -1.156714677997176) / Math.Sqrt((7.338631756763272 / (299)) + (0.015302274468929828 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8645000074523818 = (8.536640159523818 - 1.156714677997176) / 8.536640159523818 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2607425593013195 < 13.177893199564375.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.69542752697276 (T) = (0 -1.1839553919964079) / Math.Sqrt((19.645338747754504 / (299)) + (0.08313152161272068 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9138399263877192 = (13.741346105670598 - 1.1839553919964079) / 13.741346105670598 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2552825849121727 < 12.688620857145562.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.59433734149957 (T) = (0 -1.1649267248409458) / Math.Sqrt((19.061030406932367 / (299)) + (0.011300108860125375 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9135908056414438 = (13.48151355291041 - 1.1649267248409458) / 13.48151355291041 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 3.0142557596120434.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 60.49853788389088 (T) = (0 -0.06150630071294972) / Math.Sqrt((2.0867998380830466 / (299)) + (0.010668037315473045 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9883531043730688 = (5.280917995927484 - 0.06150630071294972) / 5.280917995927484 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.008499427953375 < 12.200240462572333.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.18225333753618 (T) = (0 -1.1943895189038778) / Math.Sqrt((12.149120538531719 / (299)) + (0.02261998615788988 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8916512743659005 = (11.023567761538855 - 1.1943895189038778) / 11.023567761538855 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.24145343580470163 < 2.8804413224669596.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 60.96027127226264 (T) = (0 -0.04574682835839846) / Math.Sqrt((2.116215156546452 / (299)) + (0.006547107208638495 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9913298711618083 = (5.276372382943635 - 0.04574682835839846) / 5.276372382943635 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2785037194286992 < 12.699459842611233.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.30450912903664 (T) = (0 -1.1570780116254527) / Math.Sqrt((19.30871780346404 / (299)) + (0.01539690385180379 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9142634906946965 = (13.49574435676119 - 1.1570780116254527) / 13.49574435676119 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1415479096877115 < 12.131025700419793.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.48088044790336 (T) = (0 -1.2527202043869916) / Math.Sqrt((12.36758837100962 / (299)) + (0.06951049634257774 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8865827520373163 = (11.045235419564744 - 1.2527202043869916) / 11.045235419564744 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3419837629583091 < 12.74601968894984.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.26070366494885 (T) = (0 -1.16372076271833) / Math.Sqrt((16.06645746277327 / (299)) + (0.01971944074294859 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.908175978112545 = (12.673380437906072 - 1.16372076271833) / 12.673380437906072 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 13.12 ns 0.97 ns 0.07 0.14 False
OnesComplementBenchmark - Duration of single invocation 11.07 ns 1.12 ns 0.10 0.16 False
BitwiseOrBenchmark - Duration of single invocation 13.40 ns 1.22 ns 0.09 0.16 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.22 ns 1.17 ns 0.09 0.15 False
UnaryNegateOperatorBenchmark - Duration of single invocation 15.53 ns 1.07 ns 0.07 0.10 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.21 ns 1.26 ns 0.10 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 17.43 ns 1.09 ns 0.06 0.09 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.07 ns 1.01 ns 0.08 0.12 False
DivisionOperatorBenchmark - Duration of single invocation 19.24 ns 1.01 ns 0.05 0.08 True
OnesComplementOperatorBenchmark - Duration of single invocation 11.14 ns 0.92 ns 0.08 0.15 False
AddOperatorBenchmark - Duration of single invocation 17.12 ns 1.24 ns 0.07 0.09 False
XorBenchmark - Duration of single invocation 13.65 ns 1.22 ns 0.09 0.11 False
MultiplyBenchmark - Duration of single invocation 17.60 ns 1.29 ns 0.07 0.06 True
DivideBenchmark - Duration of single invocation 19.00 ns 1.12 ns 0.06 0.06 True
AddBenchmark - Duration of single invocation 17.49 ns 1.10 ns 0.06 0.12 False
NegateBenchmark - Duration of single invocation 15.58 ns 1.15 ns 0.07 0.11 False
InequalityOperatorBenchmark - Duration of single invocation 3.79 ns 0.05 ns 0.01 0.23 False
AllBitsSetBenchmark - Duration of single invocation 6.53 ns 1.07 ns 0.16 0.18 True
MultiplyOperatorBenchmark - Duration of single invocation 17.70 ns 1.28 ns 0.07 0.08 True
EqualityOperatorBenchmark - Duration of single invocation 3.86 ns 0.00 ns 0.00 0.28 False
SubtractBenchmark - Duration of single invocation 17.39 ns 2.32 ns 0.13 0.13 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9729885026372219 < 12.65802811259805.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.32651493400949 (T) = (0 -1.2407474861057903) / Math.Sqrt((19.194905551391237 / (299)) + (0.05785734671754705 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9079227848171996 = (13.475076148236466 - 1.2407474861057903) / 13.475076148236466 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1231144242211513 < 10.78516749473044.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.97486130683023 (T) = (0 -1.201698790285761) / Math.Sqrt((7.177766854409304 / (299)) + (0.032554185876554766 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8591532355327159 = (8.531958791036988 - 1.201698790285761) / 8.531958791036988 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2218597044005948 < 12.722411126862218.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.221719465902424 (T) = (0 -1.2301708641318907) / Math.Sqrt((18.937564733099762 / (299)) + (0.023180416384141433 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9086201445465162 = (13.462166885983965 - 1.2301708641318907) / 13.462166885983965 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1738851884037944 < 12.71271369496759.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.600940036636736 (T) = (0 -1.1489410781810991) / Math.Sqrt((16.020988834838764 / (299)) + (0.012254079018403486 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9094422933851838 = (12.687391511228045 - 1.1489410781810991) / 12.687391511228045 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.070264951246614 < 15.0211082477955.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.200830896971425 (T) = (0 -1.1636387455971422) / Math.Sqrt((19.292089664204486 / (299)) + (0.00936276395485833 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9150665666539912 = (13.700597041174758 - 1.1636387455971422) / 13.700597041174758 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2595305310802154 < 12.67931242160061.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.11499677311157 (T) = (0 -1.1827360580663875) / Math.Sqrt((16.175617745858798 / (299)) + (0.013368995044751131 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9066337162909975 = (12.66769984925883 - 1.1827360580663875) / 12.66769984925883 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.086209294709002 < 16.64024808868696.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.57503529276981 (T) = (0 -1.1616274679891545) / Math.Sqrt((29.072396678302972 / (299)) + (0.007913206882862834 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9302219859125939 = (16.64747102911331 - 1.1616274679891545) / 16.64747102911331 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0129075841759334 < 12.796097024914062.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.08975736267786 (T) = (0 -1.1508248934755219) / Math.Sqrt((16.17759666843246 / (299)) + (0.02038825366798439 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9091160004114752 = (12.662568754520644 - 1.1508248934755219) / 12.662568754520644 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.014678580486834 < 18.283521239181468.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 399.4092589806953 (T) = (0 -1.1958355048114861) / Math.Sqrt((0.47802010587075044 / (299)) + (0.015798103210136735 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9410651018621702 = (20.290787675831936 - 1.1958355048114861) / 20.290787675831936 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9217106988553415 < 10.581337006921796.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.02080821207384 (T) = (0 -1.1765006072857809) / Math.Sqrt((7.189187250302381 / (299)) + (0.011498096521419077 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8622853315946623 = (8.543030462252341 - 1.1765006072857809) / 8.543030462252341 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2395890196008665 < 16.47531445373708.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.329119937096976 (T) = (0 -1.1654460169642396) / Math.Sqrt((29.300837830846422 / (299)) + (0.011011802720265154 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9299833658928822 = (16.64527339576528 - 1.1654460169642396) / 16.64527339576528 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2226156316748156 < 12.779609331022474.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.45434190698786 (T) = (0 -1.1766416497654584) / Math.Sqrt((19.136457903610836 / (299)) + (0.00856591465703581 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9126504139262704 = (13.470489130564244 - 1.1766416497654584) / 13.470489130564244 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2907083810653313 < 16.89360781459856.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 230.9855947162148 (T) = (0 -1.1488809543479659) / Math.Sqrt((1.8152702270293215 / (299)) + (0.013040494097432818 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9424648960879093 = (19.96834760398399 - 1.1488809543479659) / 19.96834760398399 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1184140596893544 < 18.423515283498542.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 252.66064378810245 (T) = (0 -1.134031097612213) / Math.Sqrt((1.7705445413499243 / (299)) + (0.009559974072034478 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.946627491994559 = (21.24747627555003 - 1.134031097612213) / 21.24747627555003 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1041976559290456 < 16.58150742517983.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.55024115497712 (T) = (0 -1.250996795763297) / Math.Sqrt((34.37897976880058 / (299)) + (0.09231364674686193 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9291366861037055 = (17.653659234651055 - 1.250996795763297) / 17.653659234651055 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1451940489441366 < 14.994229231064319.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.99060925465648 (T) = (0 -1.2371199470684635) / Math.Sqrt((19.41798200841343 / (299)) + (0.05314396295748141 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9095849454800067 = (13.682676559077892 - 1.2371199470684635) / 13.682676559077892 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.04722085432218112 < 3.6042257941348903.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 63.22436846468929 (T) = (0 -0.08015589498595728) / Math.Sqrt((2.7000614440707826 / (299)) + (0.00993227713100117 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.987134328831977 = (6.23021480489727 - 0.08015589498595728) / 6.23021480489727 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0706634722024289 < 6.428726604459288.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 218.37006818593918 (T) = (0 -1.1434632268582368) / Math.Sqrt((0.06621176064790855 / (299)) + (0.006539830780031905 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8111392329341836 = (6.054530247988185 - 1.1434632268582368) / 6.054530247988185 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2837900225168803 < 16.997584305798.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 316.373806187503 (T) = (0 -1.2167452023979803) / Math.Sqrt((0.5096698202583623 / (299)) + (0.03300058187055803 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9357689293724727 = (18.94324959105575 - 1.2167452023979803) / 18.94324959105575 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 3.580475817545534.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 64.365303621621 (T) = (0 -0.043951627008119144) / Math.Sqrt((2.735810072169402 / (299)) + (0.004184194398754577 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9929808972970804 = (6.261715901355479 - 0.043951627008119144) / 6.261715901355479 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Double>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.317291811188187 < 16.558290445215427.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.254922038393005 (T) = (0 -1.1976711876203) / Math.Sqrt((33.91199706028163 / (299)) + (0.07189716593729742 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9322246775260764 = (17.671198659085718 - 1.1976711876203) / 17.671198659085718 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 35.81 ns 1.05 ns 0.03 0.12 False
XorBenchmark - Duration of single invocation 13.24 ns 0.99 ns 0.07 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.17 ns 1.00 ns 0.08 0.13 False
BitwiseAndBenchmark - Duration of single invocation 13.09 ns 1.05 ns 0.08 0.13 False
AddOperatorBenchmark - Duration of single invocation 35.24 ns 1.02 ns 0.03 0.13 False
NegateBenchmark - Duration of single invocation 32.94 ns 1.09 ns 0.03 0.10 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.10 ns 0.86 ns 0.08 0.12 False
AddBenchmark - Duration of single invocation 35.87 ns 0.87 ns 0.02 0.12 False
OnesComplementBenchmark - Duration of single invocation 11.15 ns 1.05 ns 0.09 0.13 False
EqualityOperatorBenchmark - Duration of single invocation 3.10 ns 0.07 ns 0.02 0.28 False
MultiplyOperatorBenchmark - Duration of single invocation 35.39 ns 1.05 ns 0.03 0.05 True
InequalityOperatorBenchmark - Duration of single invocation 3.25 ns 0.00 ns 0.00 0.26 False
MultiplyBenchmark - Duration of single invocation 36.13 ns 1.04 ns 0.03 0.06 True
BitwiseOrBenchmark - Duration of single invocation 13.49 ns 0.98 ns 0.07 0.12 False
SubtractBenchmark - Duration of single invocation 36.42 ns 1.09 ns 0.03 0.11 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.11 ns 1.06 ns 0.08 0.17 False
AllBitsSetBenchmark - Duration of single invocation 6.60 ns 1.01 ns 0.15 0.14 True
UnaryNegateOperatorBenchmark - Duration of single invocation 32.52 ns 1.03 ns 0.03 0.08 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 15.20 ns 1.11 ns 0.07 0.13 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0492346790656888 < 34.184408764661754.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.701267726729725 (T) = (0 -1.1822902657516956) / Math.Sqrt((116.6798519039879 / (299)) + (0.01797296794367488 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9633524361878512 = (32.26108758039086 - 1.1822902657516956) / 32.26108758039086 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9877964276330589 < 12.765195167770203.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.59063881710688 (T) = (0 -1.1367561316419708) / Math.Sqrt((19.181233025943822 / (299)) + (0.009526541195579108 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9156928633825375 = (13.483510142206802 - 1.1367561316419708) / 13.483510142206802 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0021513187935298 < 12.721034381813922.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.25665536163871 (T) = (0 -1.2042625299152714) / Math.Sqrt((16.111980354617437 / (299)) + (0.05987427989156003 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9049791620051761 = (12.673667748340337 - 1.2042625299152714) / 12.673667748340337 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0456739761030514 < 12.709015840558724.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.819846377226504 (T) = (0 -1.1282364429434464) / Math.Sqrt((18.968921380012045 / (299)) + (0.013711949805995487 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9163178052492524 = (13.482395464219922 - 1.1282364429434464) / 13.482395464219922 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0180944361613646 < 33.88609211970769.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.65111702915077 (T) = (0 -1.1703563124905418) / Math.Sqrt((116.56912533736943 / (299)) + (0.07422354803022112 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9637661465680815 = (32.300078562981945 - 1.1703563124905418) / 32.300078562981945 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0853432672487031 < 31.10792510652953.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.97402399523145 (T) = (0 -1.248537901944077) / Math.Sqrt((97.65828936730834 / (299)) + (0.10996330517452352 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.957592548585964 = (29.441474559606185 - 1.248537901944077) / 29.441474559606185 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.8594402457700292 < 10.660423988515415.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.88804140514971 (T) = (0 -1.1653020292872818) / Math.Sqrt((7.381154184013548 / (299)) + (0.02709469413296633 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8636303706357967 = (8.545172665792775 - 1.1653020292872818) / 8.545172665792775 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.866333634594738 < 34.06474441339136.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.53688388018699 (T) = (0 -1.141140046939638) / Math.Sqrt((128.1689718598458 / (299)) + (0.018247562936123063 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9660414877778289 = (33.603947059689034 - 1.141140046939638) / 33.603947059689034 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.053606301121875 < 10.764752096755572.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.59524987036329 (T) = (0 -1.1723587010889336) / Math.Sqrt((7.251536117034407 / (299)) + (0.016376244819932648 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8626330359281835 = (8.53450252038777 - 1.1723587010889336) / 8.53450252038777 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.06762467836063357 < 3.0087103950301204.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 61.44521122885057 (T) = (0 -0.08259834737797851) / Math.Sqrt((2.187723249525882 / (299)) + (0.007882768780222179 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9848726671788923 = (5.460205599676283 - 0.08259834737797851) / 5.460205599676283 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0543938246374893 < 33.74370013264642.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 750.4944385130398 (T) = (0 -1.1576582937120428) / Math.Sqrt((0.4751967348521321 / (299)) + (0.014002168970654049 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9681468259647273 = (36.343577328592445 - 1.1576582937120428) / 36.343577328592445 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.0008151865349065017 < 3.038454635560563.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 61.81752792891952 (T) = (0 -0.06832174805878707) / Math.Sqrt((2.1930542116560647 / (299)) + (0.009542077472038108 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9876008508952349 = (5.510196504736814 - 0.06832174805878707) / 5.510196504736814 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.041544218632171 < 34.22497328621521.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 292.35706445135577 (T) = (0 -1.2371634852342834) / Math.Sqrt((2.2408925331199465 / (299)) + (0.1860569243756327 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9672134597094758 = (37.73388330307732 - 1.2371634852342834) / 37.73388330307732 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.975116571953757 < 12.683203602760662.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.113685964708 (T) = (0 -1.1780957180301712) / Math.Sqrt((19.093510997330615 / (299)) + (0.10081597520501058 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9126405429316566 = (13.485611719272931 - 1.1780957180301712) / 13.485611719272931 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.087787022544401 < 34.258402920207374.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.50818141869075 (T) = (0 -1.1650105119395868) / Math.Sqrt((128.1971345511906 / (299)) + (0.012552346375574248 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.965330382321514 = (33.60321197491963 - 1.1650105119395868) / 33.60321197491963 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0640803996105122 < 12.655178238555314.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.20350874807529 (T) = (0 -1.15800570385832) / Math.Sqrt((16.129763697475244 / (299)) + (0.01441704981289387 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9084965372343543 = (12.655321108712016 - 1.15800570385832) / 12.655321108712016 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.014480870203742 < 6.396484465865878.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 152.68958889602524 (T) = (0 -1.16638739223017) / Math.Sqrt((0.05579402073396114 / (299)) + (0.01848153354692794 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8074894858104393 = (6.058824356375957 - 1.16638739223017) / 6.058824356375957 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0345533047718205 < 31.112084722105113.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.38325950872839 (T) = (0 -1.1840969489385562) / Math.Sqrt((97.65466212983556 / (299)) + (0.014481002220241174 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.959770395560037 = (29.43347232522888 - 1.1840969489385562) / 29.43347232522888 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1058196125158373 < 12.749837902575834.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.3093758975007 (T) = (0 -1.1382575004425852) / Math.Sqrt((16.229035732959392 / (299)) + (0.011312840466557318 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9102184747346377 = (12.678081566093914 - 1.1382575004425852) / 12.678081566093914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 22.69 ns 1.22 ns 0.05 0.12 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.54 ns 0.98 ns 0.10 0.15 False
SubtractBenchmark - Duration of single invocation 22.67 ns 1.30 ns 0.06 0.12 False
AndNotBenchmark - Duration of single invocation 20.25 ns 1.07 ns 0.05 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.49 ns 0.95 ns 0.10 0.15 False
MultiplyBenchmark - Duration of single invocation 23.60 ns 1.14 ns 0.05 0.05 True
UnaryNegateOperatorBenchmark - Duration of single invocation 22.65 ns 1.01 ns 0.04 0.11 False
OnesComplementOperatorBenchmark - Duration of single invocation 8.91 ns 1.10 ns 0.12 0.15 False
MultiplyOperatorBenchmark - Duration of single invocation 23.41 ns 1.31 ns 0.06 0.05 True
AddOperatorBenchmark - Duration of single invocation 22.62 ns 1.25 ns 0.06 0.12 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.79 ns 1.57 ns 0.16 0.16 False
OnesComplementBenchmark - Duration of single invocation 8.78 ns 1.13 ns 0.13 0.14 False
NegateBenchmark - Duration of single invocation 22.68 ns 1.03 ns 0.05 0.12 False
AddBenchmark - Duration of single invocation 22.82 ns 2.75 ns 0.12 0.11 False
ConditionalSelectBenchmark - Duration of single invocation 32.79 ns 1.20 ns 0.04 0.41 False
OneBenchmark - Duration of single invocation 12.34 ns 1.10 ns 0.09 0.21 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2232188138058504 < 21.630247294905256.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.608682411695675 (T) = (0 -1.2650850392433763) / Math.Sqrt((49.89463636493455 / (299)) + (0.023720936837456977 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9414115165796944 = (21.592725487837466 - 1.2650850392433763) / 21.592725487837466 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9771047876483431 < 9.07163352694982.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.87882395747947 (T) = (0 -1.2486329367020763) / Math.Sqrt((16.144012102806208 / (299)) + (0.02503227589475132 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8896194670654144 = (11.312075630600999 - 1.2486329367020763) / 11.312075630600999 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.295631756113688 < 21.676428205182145.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.87723423986376 (T) = (0 -1.3122839600333756) / Math.Sqrt((57.10518168525548 / (299)) + (0.049175189815186114 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9424231903278643 = (22.791883876616655 - 1.3122839600333756) / 22.791883876616655 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.071778581491835 < 16.911995028923112.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.07807904482796 (T) = (0 -1.249717058727427) / Math.Sqrt((51.59951899339555 / (299)) + (0.022047754230452243 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9376014522616798 = (20.02798308653506 - 1.249717058727427) / 20.02798308653506 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9515607757019794 < 9.060747556578495.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.97949877041836 (T) = (0 -1.2498039219659833) / Math.Sqrt((16.103356963874106 / (299)) + (0.026804296146790033 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8897051226079318 = (11.331477503921331 - 1.2498039219659833) / 11.331477503921331 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1426691966238487 < 22.4424615613354.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 202.81844953575757 (T) = (0 -1.3631958840075806) / Math.Sqrt((2.0786369596880525 / (299)) + (0.17471594072400032 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9478003174181294 = (26.11502247871964 - 1.3631958840075806) / 26.11502247871964 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0097573430349243 < 21.52275095135952.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.15022884166462 (T) = (0 -1.1835061319414173) / Math.Sqrt((50.353506711077124 / (299)) + (0.024195297825597473 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9447382077544336 = (21.416354480185547 - 1.1835061319414173) / 21.416354480185547 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.097268822204067 < 8.616585719155353.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.59832621117554 (T) = (0 -1.1623955716869236) / Math.Sqrt((5.565581032064656 / (299)) + (0.018826147196340282 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8454020065319416 = (7.518827027513053 - 1.1623955716869236) / 7.518827027513053 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3127992213985924 < 22.37245622614425.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 496.1114278211036 (T) = (0 -1.2244265120231965) / Math.Sqrt((0.37431055839061333 / (299)) + (0.023225823694332834 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9506640258752695 = (24.81812782144771 - 1.2244265120231965) / 24.81812782144771 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2528257762092696 < 21.61621284842872.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.11558347209156 (T) = (0 -1.3079736569101563) / Math.Sqrt((50.246030739792886 / (299)) + (0.05384976569327884 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9394128114322743 = (21.58828768639883 - 1.3079736569101563) / 21.58828768639883 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.573203498477966 < 9.140504230725323.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.90722193104904 (T) = (0 -1.2514484147152836) / Math.Sqrt((16.135485486207635 / (299)) + (0.029311376164735746 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8896035274779602 = (11.335945670414814 - 1.2514484147152836) / 11.335945670414814 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1327388320055163 < 8.614408802126556.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.32582428404166 (T) = (0 -1.2199369810360958) / Math.Sqrt((5.526775723287009 / (299)) + (0.01798249246630846 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8375855857255261 = (7.5112605398092995 - 1.2199369810360958) / 7.5112605398092995 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0320142078267176 < 21.53497702098007.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.04953603218536 (T) = (0 -1.193429071337644) / Math.Sqrt((53.4434874020244 / (299)) + (0.019371106492886463 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9457020480755848 = (21.97926494536924 - 1.193429071337644) / 21.97926494536924 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.7533582026405536 < 21.679356757873173.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.23601042682327 (T) = (0 -1.3377860904399388) / Math.Sqrt((57.376819883211866 / (299)) + (0.12894407917210424 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9412593346139922 = (22.77444563572484 - 1.3377860904399388) / 22.77444563572484 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2030029417113695 < 31.128962573209023.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.36719238862814 (T) = (0 -1.1743441556286516) / Math.Sqrt((363.7275933897926 / (299)) + (0.02367275070858895 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9749174880982817 = (46.81924044250933 - 1.1743441556286516) / 46.81924044250933 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt16>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1034465523908152 < 11.68713949843272.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 31.991877401723208 (T) = (0 -1.2472653248018295) / Math.Sqrt((32.84591273216309 / (299)) + (0.010244302239615154 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8949417505501267 = (11.872131235129153 - 1.2472653248018295) / 11.872131235129153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllBitsSetBenchmark - Duration of single invocation 6.56 ns 1.01 ns 0.15 0.17 True
MultiplyOperatorBenchmark - Duration of single invocation 54.58 ns 1.05 ns 0.02 0.06 True
BitwiseOrOperatorBenchmark - Duration of single invocation 13.08 ns 1.24 ns 0.09 0.13 False
SubtractBenchmark - Duration of single invocation 51.50 ns 1.08 ns 0.02 0.11 False
XorBenchmark - Duration of single invocation 13.22 ns 1.02 ns 0.08 0.14 False
UnaryNegateOperatorBenchmark - Duration of single invocation 48.78 ns 1.05 ns 0.02 0.11 False
MultiplyBenchmark - Duration of single invocation 53.87 ns 1.02 ns 0.02 0.04 True
BitwiseOrBenchmark - Duration of single invocation 13.25 ns 1.08 ns 0.08 0.13 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.75 ns 1.10 ns 0.09 0.12 False
InequalityOperatorBenchmark - Duration of single invocation 3.24 ns 0.12 ns 0.04 0.32 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.11 ns 1.07 ns 0.08 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 14.43 ns 1.07 ns 0.07 0.11 False
AddOperatorBenchmark - Duration of single invocation 52.26 ns 1.05 ns 0.02 0.13 False
OnesComplementBenchmark - Duration of single invocation 10.85 ns 1.03 ns 0.09 0.17 False
AddBenchmark - Duration of single invocation 52.38 ns 0.97 ns 0.02 0.11 False
NegateBenchmark - Duration of single invocation 48.61 ns 0.99 ns 0.02 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 52.98 ns 0.95 ns 0.02 0.13 False
BitwiseAndBenchmark - Duration of single invocation 13.08 ns 0.94 ns 0.07 0.12 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0095445239590723 < 6.33176333358847.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 192.346807625689 (T) = (0 -1.149058151440351) / Math.Sqrt((0.08223974767396725 / (299)) + (0.0085905574635462 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8099942941188158 = (6.047492869287248 - 1.149058151440351) / 6.047492869287248 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.047319803057948 < 51.494664382638085.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 375.06553614829227 (T) = (0 -1.2253649765211398) / Math.Sqrt((4.2743609290196245 / (299)) + (0.14397530889545496 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.977720224664152 = (54.998982622124444 - 1.2253649765211398) / 54.998982622124444 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2423680623421485 < 12.685863682322816.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.69611307467224 (T) = (0 -1.1799325629835922) / Math.Sqrt((15.976205338460115 / (299)) + (0.05216848592805189 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.906969536893482 = (12.683292370936524 - 1.1799325629835922) / 12.683292370936524 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0791376490611073 < 49.56631956834375.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.53485164675762 (T) = (0 -1.1466691650254461) / Math.Sqrt((273.65169500665627 / (299)) + (0.04023909524934085 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9763965584226235 = (48.58059199826645 - 1.1466691650254461) / 48.58059199826645 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.020709124147049 < 12.733187620555876.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.63155365649772 (T) = (0 -1.1440738962076247) / Math.Sqrt((19.088582854246713 / (299)) + (0.007229143285317427 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9150142552827744 = (13.461950589646758 - 1.1440738962076247) / 13.461950589646758 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0528303012693607 < 46.36839402564059.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.50662978441437 (T) = (0 -1.141403489812712) / Math.Sqrt((228.04989700611847 / (299)) + (0.02893261166457189 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9743001026375844 = (44.41276452262941 - 1.141403489812712) / 44.41276452262941 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0221937243027404 < 51.25034506592946.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 430.7241743159972 (T) = (0 -1.140155557620321) / Math.Sqrt((4.592502151329993 / (299)) + (0.010098895469319687 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.97937451988618 = (55.27898266263181 - 1.140155557620321) / 55.27898266263181 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0786443914991988 < 12.656309474380818.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.031615155521386 (T) = (0 -1.1863484447112127) / Math.Sqrt((19.16293265926422 / (299)) + (0.091117849972964 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9118307682540697 = (13.455356491364363 - 1.1863484447112127) / 13.455356491364363 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1009064519008673 < 10.821494720533375.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.64104108486708 (T) = (0 -1.1355910526566044) / Math.Sqrt((7.43927133593022 / (299)) + (0.008163296878311096 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8671387935687153 = (8.547198111165184 - 1.1355910526566044) / 8.547198111165184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.11890893940948669 < 3.034979854526752.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 60.28948916866736 (T) = (0 -0.08356185701256248) / Math.Sqrt((2.286347958677136 / (299)) + (0.011992595692232115 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9848958198615386 = (5.532366288440942 - 0.08356185701256248) / 5.532366288440942 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0657999644386877 < 12.580367419693737.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.05651861685467 (T) = (0 -1.1481842631701462) / Math.Sqrt((16.31665811720869 / (299)) + (0.010561829245866433 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9092781360264456 = (12.656092069546133 - 1.1481842631701462) / 12.656092069546133 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0741227396065207 < 12.770647358575532.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.31621217483518 (T) = (0 -1.1353827479169307) / Math.Sqrt((16.199890857175873 / (299)) + (0.014247808361772862 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9104586407840125 = (12.679981160194501 - 1.1353827479169307) / 12.679981160194501 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0548946470769105 < 49.98814364685258.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.54622441126835 (T) = (0 -1.1386227849164527) / Math.Sqrt((271.35300279307984 / (299)) + (0.0073702906866982745 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9764489398894396 = (48.3469864868583 - 1.1386227849164527) / 48.3469864868583 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.027914428676164 < 10.756980855849184.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.181298616862954 (T) = (0 -1.2165534779798917) / Math.Sqrt((7.37591647816435 / (299)) + (0.08901297865184533 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8574428966479859 = (8.533797680890546 - 1.2165534779798917) / 8.533797680890546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9693183539020012 < 49.33769293040486.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.4705505720009 (T) = (0 -1.1591658387544306) / Math.Sqrt((274.6854191839709 / (299)) + (0.018421068445008296 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.976147021045805 = (48.596271391526464 - 1.1591658387544306) / 48.596271391526464 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9919693964639675 < 46.8899769263761.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.50313155405929 (T) = (0 -1.1822464903552825) / Math.Sqrt((227.95385178191748 / (299)) + (0.015595983456108619 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9733878264400796 = (44.42502555055564 - 1.1822464903552825) / 44.42502555055564 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9526892083150006 < 49.87601142871293.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.532442249867024 (T) = (0 -1.1507825147527446) / Math.Sqrt((271.26888558981557 / (299)) + (0.06894629589908018 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9762275533739609 = (48.40824896383356 - 1.1507825147527446) / 48.40824896383356 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Byte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9376496436286214 < 12.757188432667721.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.447997821789905 (T) = (0 -1.1455994817091624) / Math.Sqrt((19.164298621820176 / (299)) + (0.013079737146739268 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9149229417873651 = (13.465433640711236 - 1.1455994817091624) / 13.465433640711236 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EqualityOperatorBenchmark - Duration of single invocation 3.37 ns 0.00 ns 0.00 0.39 False
SubtractionOperatorBenchmark - Duration of single invocation 52.22 ns 1.11 ns 0.02 0.13 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.16 ns 1.01 ns 0.09 0.17 False
AddBenchmark - Duration of single invocation 52.02 ns 1.28 ns 0.02 0.10 False
BitwiseOrBenchmark - Duration of single invocation 13.20 ns 1.03 ns 0.08 0.15 False
InequalityOperatorBenchmark - Duration of single invocation 3.28 ns 0.04 ns 0.01 0.38 False
OnesComplementBenchmark - Duration of single invocation 11.19 ns 1.28 ns 0.11 0.16 False
NegateBenchmark - Duration of single invocation 48.92 ns 1.23 ns 0.03 0.11 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.27 ns 1.13 ns 0.09 0.13 False
XorBenchmark - Duration of single invocation 13.17 ns 1.18 ns 0.09 0.14 False
UnaryNegateOperatorBenchmark - Duration of single invocation 48.75 ns 1.09 ns 0.02 0.11 False
AllBitsSetBenchmark - Duration of single invocation 6.54 ns 0.92 ns 0.14 0.17 True
MultiplyBenchmark - Duration of single invocation 54.02 ns 1.00 ns 0.02 0.04 True
BitwiseAndOperatorBenchmark - Duration of single invocation 13.16 ns 1.01 ns 0.08 0.13 False
SubtractBenchmark - Duration of single invocation 51.46 ns 1.27 ns 0.02 0.11 False
BitwiseAndBenchmark - Duration of single invocation 13.17 ns 1.06 ns 0.08 0.13 False
MultiplyOperatorBenchmark - Duration of single invocation 54.84 ns 0.99 ns 0.02 0.06 True
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.14 ns 0.96 ns 0.07 0.12 False
AddOperatorBenchmark - Duration of single invocation 51.78 ns 1.19 ns 0.02 0.12 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 2.970590264188213.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 62.26096258697371 (T) = (0 -0.03001913598675415) / Math.Sqrt((2.252129481603066 / (299)) + (0.004362087300176503 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9945431129429303 = (5.501146656107347 - 0.03001913598675415) / 5.501146656107347 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1109356053949484 < 49.584629086518355.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.415538745480575 (T) = (0 -1.1894138527734242) / Math.Sqrt((272.2498188882269 / (299)) + (0.034826642074078554 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9754161232292291 = (48.38186685785807 - 1.1894138527734242) / 48.38186685785807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0121767085477964 < 10.774080580384299.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.45409913559951 (T) = (0 -1.176163988839476) / Math.Sqrt((7.2959683174300265 / (299)) + (0.014547453396560286 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8621204756546963 = (8.530374574646096 - 1.176163988839476) / 8.530374574646096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2799236307873545 < 49.445510250254735.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.42772464992511 (T) = (0 -1.208586555034839) / Math.Sqrt((274.07654471451104 / (299)) + (0.02974167556423808 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9751139453236969 = (48.56481152818788 - 1.208586555034839) / 48.56481152818788 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0345637536819405 < 12.698823329944506.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.10305919231857 (T) = (0 -1.2560823567919015) / Math.Sqrt((19.212896956764546 / (299)) + (0.06909161290969493 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9067644964811755 = (13.472146439776516 - 1.2560823567919015) / 13.472146439776516 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.04085178329861341 < 3.104567366542147.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 61.10605939347357 (T) = (0 -0.06134496754334595) / Math.Sqrt((2.2711641420867954 / (299)) + (0.009458462294835737 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9889053994231076 = (5.529263277049743 - 0.06134496754334595) / 5.529263277049743 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2775403682814694 < 10.729606784943838.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.28326714276443 (T) = (0 -1.2059177455596226) / Math.Sqrt((7.286629522207985 / (299)) + (0.015806674225848943 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8587402898151928 = (8.536883899747105 - 1.2059177455596226) / 8.536883899747105 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2342689385066323 < 46.78929522642719.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.35713116637932 (T) = (0 -1.2526815516079102) / Math.Sqrt((228.04450812563186 / (299)) + (0.046275816901059814 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9717954398927158 = (44.414149585846 - 1.2526815516079102) / 44.414149585846 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1303152384100403 < 12.648402310273669.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.246757341507546 (T) = (0 -1.164855588498392) / Math.Sqrt((16.158066606018053 / (299)) + (0.0098975221533152 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9079936343667634 = (12.660597780177916 - 1.164855588498392) / 12.660597780177916 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1810057597076746 < 12.683144509878447.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.05590748930666 (T) = (0 -1.2399115094120676) / Math.Sqrt((19.24830514105451 / (299)) + (0.07491490567271888 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9080007762541975 = (13.477412731633393 - 1.2399115094120676) / 13.477412731633393 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.085483441731322 < 46.745077637588444.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.51407317438172 (T) = (0 -1.1623021775360127) / Math.Sqrt((227.9045536045062 / (299)) + (0.013401115047869415 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9738263034517735 = (44.40726113693591 - 1.1623021775360127) / 44.40726113693591 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9156393509555869 < 6.386909657377067.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 153.2573797135237 (T) = (0 -1.1548563693642464) / Math.Sqrt((0.06618725228570985 / (299)) + (0.01841722647472356 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8092586359196072 = (6.054567004551269 - 1.1548563693642464) / 6.054567004551269 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9997371884168837 < 51.33652767616804.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 358.03531856279255 (T) = (0 -1.2716494589506862) / Math.Sqrt((4.5490608640252015 / (299)) + (0.1725704735038767 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9769777524037422 = (55.23567816885903 - 1.2716494589506862) / 55.23567816885903 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0098639336451367 < 12.803306273528534.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.430034794062756 (T) = (0 -1.1672736900348315) / Math.Sqrt((16.0723521161219 / (299)) + (0.010861000392277541 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9079275736134719 = (12.677777004969034 - 1.1672736900348315) / 12.677777004969034 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2695152599941177 < 49.43270174017141.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.41130899244305 (T) = (0 -1.2064010177152602) / Math.Sqrt((274.4877779420455 / (299)) + (0.012543330653028121 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9751580617186951 = (48.56307925952596 - 1.2064010177152602) / 48.56307925952596 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0585264023851575 < 12.710088429947763.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.36179527017481 (T) = (0 -1.158505480060848) / Math.Sqrt((19.044410072177556 / (299)) + (0.025154738236612052 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9139822122990696 = (13.468208274418535 - 1.158505480060848) / 13.468208274418535 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9883009157255475 < 51.19181424247223.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 422.5496969362631 (T) = (0 -1.1690759181245358) / Math.Sqrt((4.354876964850811 / (299)) + (0.03847755080251955 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.978749284969565 = (55.01348620270908 - 1.1690759181245358) / 55.01348620270908 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9591988604981613 < 12.710738539931896.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.840241950823206 (T) = (0 -1.1747099711549767) / Math.Sqrt((16.14184036708218 / (299)) + (0.03289399338028257 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9072989519063773 = (12.672024699964435 - 1.1747099711549767) / 12.672024699964435 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<SByte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1861151243935821 < 49.96488783029815.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.59343744658832 (T) = (0 -1.1499711144049756) / Math.Sqrt((271.2820334510703 / (299)) + (0.009775855863725973 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9762401998435165 = (48.39986476448449 - 1.1499711144049756) / 48.39986476448449 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 7.32 ns 0.97 ns 0.13 0.10 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.54 ns 1.02 ns 0.11 0.14 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.28 ns 1.06 ns 0.11 0.15 False
AddBenchmark - Duration of single invocation 7.20 ns 1.08 ns 0.15 0.13 False
OnesComplementOperatorBenchmark - Duration of single invocation 8.98 ns 1.03 ns 0.12 0.18 False
DivideBenchmark - Duration of single invocation 8.83 ns 1.08 ns 0.12 0.07 False
DivisionOperatorBenchmark - Duration of single invocation 9.28 ns 1.12 ns 0.12 0.09 True
MultiplyBenchmark - Duration of single invocation 7.79 ns 1.00 ns 0.13 0.11 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.62 ns 0.99 ns 0.10 0.16 False
SubtractBenchmark - Duration of single invocation 7.18 ns 1.01 ns 0.14 0.15 False
AddOperatorBenchmark - Duration of single invocation 7.28 ns 0.99 ns 0.14 0.13 False
NegateBenchmark - Duration of single invocation 7.36 ns 1.00 ns 0.14 0.12 False
ConditionalSelectBenchmark - Duration of single invocation 32.72 ns 0.98 ns 0.03 0.39 False
MultiplyOperatorBenchmark - Duration of single invocation 7.58 ns 1.26 ns 0.17 0.09 True
OnesComplementBenchmark - Duration of single invocation 9.03 ns 1.10 ns 0.12 0.13 False
SubtractionOperatorBenchmark - Duration of single invocation 7.24 ns 1.02 ns 0.14 0.12 False
OneBenchmark - Duration of single invocation 6.34 ns 0.96 ns 0.15 0.46 False
AndNotBenchmark - Duration of single invocation 17.72 ns 0.83 ns 0.05 0.11 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Double>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9680533149868615 < 6.975677025382067.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.635877310762176 (T) = (0 -1.1250049996726992) / Math.Sqrt((6.637030115800201 / (299)) + (0.006360869029128274 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.858781752820657 = (7.966428008725931 - 1.1250049996726992) / 7.966428008725931 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0187419712895922 < 9.007696677394357.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.48793209907502 (T) = (0 -1.120783836301258) / Math.Sqrt((16.2616414489828 / (299)) + (0.014433471202433646 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9009989953981135 = (11.320933972420528 - 1.120783836301258) / 11.320933972420528 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0553865593768879 < 9.059663715336125.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.62692140915176 (T) = (0 -1.2310053723316172) / Math.Sqrt((16.158729044794757 / (299)) + (0.04652909150149152 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8912950478409248 = (11.324280521555314 - 1.2310053723316172) / 11.324280521555314 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0809222274441184 < 6.979850668280357.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.87244794856658 (T) = (0 -1.1923933690753663) / Math.Sqrt((11.337880115161328 / (299)) + (0.0586739013342307 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8761540525689757 = (9.628037039641269 - 1.1923933690753663) / 9.628037039641269 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0348909827729003 < 8.6627013583695.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.015057526113765 (T) = (0 -1.1807849285313228) / Math.Sqrt((5.535907337543723 / (299)) + (0.05116656117016065 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8429696340028248 = (7.519468741176872 - 1.1807849285313228) / 7.519468741176872 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0767844534390072 < 8.456625311895236.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 83.2680247486111 (T) = (0 -1.0982124571715624) / Math.Sqrt((4.5107611952242195 / (299)) + (0.00871406134935306 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.904113858087514 = (11.453296954776704 - 1.0982124571715624) / 11.453296954776704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1211442194138568 < 8.481340252856256.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 134.18487108298183 (T) = (0 -1.1606060059893886) / Math.Sqrt((1.1097153002740427 / (299)) + (0.012435378094365977 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8828917615307119 = (9.910541061496371 - 1.1606060059893886) / 9.910541061496371 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9981480601502696 < 7.47429270951442.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 64.18377236416295 (T) = (0 -1.2593216807773786) / Math.Sqrt((4.073154650847233 / (299)) + (0.1635781889141983 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8800816422601808 = (10.501492052698602 - 1.2593216807773786) / 10.501492052698602 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9893319071921785 < 9.061704020387273.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.75894339893306 (T) = (0 -1.10765026414981) / Math.Sqrt((16.160923270168897 / (299)) + (0.009171282203960612 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.902137773982137 = (11.318465859826526 - 1.10765026414981) / 11.318465859826526 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0128885822474871 < 7.017871365641554.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.69278245900281 (T) = (0 -1.1343362910794716) / Math.Sqrt((11.552678658124309 / (299)) + (0.01283835043278806 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8816752398490834 = (9.586635034228575 - 1.1343362910794716) / 9.586635034228575 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9932320370305845 < 6.942434234093177.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.24390601200193 (T) = (0 -1.1504370243688762) / Math.Sqrt((6.4489930312484605 / (299)) + (0.0283861627495442 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8585562473765357 = (8.133530134988998 - 1.1504370243688762) / 8.133530134988998 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9985531637160239 < 7.0078828642609015.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.67168828826344 (T) = (0 -1.165493340385015) / Math.Sqrt((9.376011191091283 / (299)) + (0.062229925031785045 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8684032369226916 = (8.85655021545119 - 1.165493340385015) / 8.85655021545119 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9813774132392183 < 31.151574892048714.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.534918384254006 (T) = (0 -1.1099418406692927) / Math.Sqrt((362.0188761444369 / (299)) + (0.014145246500402435 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9762956564611488 = (46.82440746988451 - 1.1099418406692927) / 46.82440746988451 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.261074432162496 < 7.296299387436848.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 126.99159049143307 (T) = (0 -1.1495374353462047) / Math.Sqrt((1.0336772939899852 / (299)) + (0.008081461022164704 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8720825567582499 = (8.986557315515627 - 1.1495374353462047) / 8.986557315515627 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1017034298563577 < 8.602404488304366.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.243585544897805 (T) = (0 -1.1533928353324394) / Math.Sqrt((5.5450201829062955 / (299)) + (0.008045565376051044 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8464190227323807 = (7.509998020930803 - 1.1533928353324394) / 7.509998020930803 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.016190160533396 < 6.957173304825667.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.26386563062812 (T) = (0 -1.1446932437917006) / Math.Sqrt((6.537525875134344 / (299)) + (0.020922391724904784 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8591397497676615 = (8.126446189777562 - 1.1446932437917006) / 8.126446189777562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9562755959548337 < 5.915689552464527.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 29.921344644823773 (T) = (0 -1.2437013906199648) / Math.Sqrt((10.233652231557627 / (299)) + (0.02358608801279407 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8187438462387919 = (6.8615678133524325 - 1.2437013906199648) / 6.8615678133524325 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Double>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.8347233391388879 < 16.77999793339872.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.32570700400329 (T) = (0 -1.1607182574955321) / Math.Sqrt((51.9639625233674 / (299)) + (0.17030162828206452 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.942047959338772 = (20.02894538745885 - 1.1607182574955321) / 20.02894538745885 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Convert_float_int - Duration of single invocation 25.41 μs 16.29 μs 0.64 0.01 False
Widen_short - Duration of single invocation 61.78 μs 34.25 μs 0.55 0.01 False
Narrow_short - Duration of single invocation 44.28 μs 35.36 μs 0.80 0.01 False
Narrow_long - Duration of single invocation 21.78 μs 13.39 μs 0.61 0.01 False
Widen_ushort - Duration of single invocation 61.12 μs 34.41 μs 0.56 0.11 False
Convert_long_double - Duration of single invocation 56.22 μs 49.71 μs 0.88 0.01 False
Narrow_ushort - Duration of single invocation 44.26 μs 35.39 μs 0.80 0.01 False
Widen_int - Duration of single invocation 53.27 μs 26.39 μs 0.50 0.01 False
Narrow_double - Duration of single invocation 23.07 μs 14.54 μs 0.63 0.01 False
Narrow_uint - Duration of single invocation 29.31 μs 20.56 μs 0.70 0.01 False
Widen_uint - Duration of single invocation 52.60 μs 25.67 μs 0.49 0.01 False
Convert_double_long - Duration of single invocation 18.24 μs 10.29 μs 0.56 0.01 False
Convert_ulong_double - Duration of single invocation 56.08 μs 49.48 μs 0.88 0.03 False
Widen_float - Duration of single invocation 64.09 μs 39.44 μs 0.62 0.01 False
Widen_byte - Duration of single invocation 72.90 μs 49.55 μs 0.68 0.01 False
Convert_int_float - Duration of single invocation 26.29 μs 17.44 μs 0.66 0.01 False
Widen_sbyte - Duration of single invocation 72.67 μs 49.68 μs 0.68 0.01 False
Narrow_ulong - Duration of single invocation 21.90 μs 13.30 μs 0.61 0.01 False
Narrow_int - Duration of single invocation 28.97 μs 20.31 μs 0.70 0.01 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorConvert* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 16.28550833118113 < 24.162744106800766.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.83489631327007 (T) = (0 -16198.545755714216) / Math.Sqrt((9027205.263891865 / (299)) + (6140.202000127072 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.34934840769375664 = (24895.882753930182 - 16198.545755714216) / 24895.882753930182 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_short

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 34.246972766229035 < 58.5664415097042.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 39.76789794304391 (T) = (0 -34242.325768524875) / Math.Sqrt((55617822.36904002 / (299)) + (8242.692193215653 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.3339418679980646 = (51410.41618335106 - 34242.325768524875) / 51410.41618335106 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_short

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 35.35735500755858 < 42.22975779459919.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.246437744216664 (T) = (0 -35444.253810836104) / Math.Sqrt((7283392.9447731655 / (299)) + (11380.764891040833 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.1796835977605321 = (43208.02767575184 - 35444.253810836104) / 43208.02767575184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_long

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.39209874434069 < 20.694708297481764.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.27689895290594 (T) = (0 -13334.65620106077) / Math.Sqrt((8258811.588088463 / (299)) + (4448.1979421918095 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.3813071520105457 = (21552.950295763014 - 13334.65620106077) / 21552.950295763014 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 34.40747043956044 < 59.11217809576746.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.903531910245405 (T) = (0 -34384.3047592745) / Math.Sqrt((60149323.03764479 / (299)) + (4405.373858262346 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.3846773527613336 = (55880.122263625686 - 34384.3047592745) / 55880.122263625686 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Convert_long_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 49.70834182641108 < 53.06411965752398.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.49441535923764 (T) = (0 -50039.331140149225) / Math.Sqrt((4056645.3178655435 / (299)) + (60848.473728826146 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.0996473645074635 = (55577.48060877868 - 50039.331140149225) / 55577.48060877868 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_ushort

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 35.39001535229477 < 42.090118624848294.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.43989539189363 (T) = (0 -35471.3900278144) / Math.Sqrt((7205855.024912623 / (299)) + (4705.603085854582 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.17850388344730983 = (43179.01121269548 - 35471.3900278144) / 43179.01121269548 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 26.391712008150648 < 50.59942385096701.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.54745220313224 (T) = (0 -26290.29157684702) / Math.Sqrt((52388494.7883333 / (299)) + (5453.794453395877 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.4040105851292456 = (44112.01091977162 - 26290.29157684702) / 44112.01091977162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 14.538138475836432 < 21.692261823778246.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.04805598746905 (T) = (0 -14374.28144662862) / Math.Sqrt((14964676.647131955 / (299)) + (23961.556803399526 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.414647545498373 = (24556.626244724608 - 14374.28144662862) / 24556.626244724608 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_uint

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 20.559712028508773 < 27.773478410101568.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.01002406617386 (T) = (0 -20565.615090616408) / Math.Sqrt((8782057.354230002 / (299)) + (3378.7003389744377 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.2904977989895458 = (28985.977860713334 - 20565.615090616408) / 28985.977860713334 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_uint

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 25.670634435261704 < 50.150751353558796.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 29.2689701124354 (T) = (0 -25805.765345159805) / Math.Sqrt((63356923.97836274 / (299)) + (8493.099308322786 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.3432083548921479 = (39290.6419217958 - 25805.765345159805) / 39290.6419217958 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 10.294461904369786 < 17.385175204633494.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.350163669099516 (T) = (0 -10301.993799898228) / Math.Sqrt((11956905.89135593 / (299)) + (721.2437260839095 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.4737067339301817 = (19574.62590549574 - 10301.993799898228) / 19574.62590549574 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Convert_ulong_double

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 49.47961801201672 < 53.28061846699522.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.94970514823177 (T) = (0 -49370.691448731865) / Math.Sqrt((5773136.850769505 / (299)) + (42689.371364532155 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.11465671179710121 = (55764.46120571631 - 49370.691448731865) / 55764.46120571631 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_float

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 39.43773002083333 < 60.8770006147541.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.6386197162067 (T) = (0 -39464.303674353636) / Math.Sqrt((91200705.64285205 / (299)) + (34263.91595757517 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.4005900510300692 = (65838.5863333966 - 39464.303674353636) / 65838.5863333966 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_byte

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 49.55470714155307 < 70.05187064103659.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.52936186178807 (T) = (0 -49742.983365598986) / Math.Sqrt((52768417.85699896 / (299)) + (39172.6928038918 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.29592747544239895 = (70650.36857794534 - 49742.983365598986) / 70650.36857794534 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 17.43937959259259 < 25.02077278646175.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.54994628518143 (T) = (0 -17406.05296025321) / Math.Sqrt((10686610.841475192 / (299)) + (2082.235317388982 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.34554363008059563 = (26596.200694626517 - 17406.05296025321) / 26596.200694626517 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 49.68173034018987 < 69.71394450958326.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.8255604794073 (T) = (0 -49476.598347142346) / Math.Sqrt((53521619.98870557 / (299)) + (189031.51522278623 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.2994365605502063 = (70624.00856373567 - 49476.598347142346) / 70624.00856373567 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 13.299431800158604 < 20.81139085972426.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.74747918727588 (T) = (0 -13294.25448378589) / Math.Sqrt((8049523.003241178 / (299)) + (15402.780960107191 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.3785262513955116 = (21391.498053840525 - 13294.25448378589) / 21391.498053840525 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorConvert.Narrow_int

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 20.311872639973956 < 27.38372430537659.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.87020352285725 (T) = (0 -20337.403076366187) / Math.Sqrt((9111215.2158979 / (299)) + (2774.758797520369 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.2959233110479144 = (28885.210085048286 - 20337.403076366187) / 28885.210085048286 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Float

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 37.82 ns 1.26 ns 0.03 0.17 False
AllBitsSetBenchmark - Duration of single invocation 6.58 ns 1.08 ns 0.16 0.09 False
NegateBenchmark - Duration of single invocation 35.51 ns 0.96 ns 0.03 0.16 False
EqualityOperatorBenchmark - Duration of single invocation 7.63 ns 0.00 ns 0.00 0.20 False
DivideBenchmark - Duration of single invocation 41.35 ns 1.04 ns 0.03 0.12 False
OnesComplementOperatorBenchmark - Duration of single invocation 12.98 ns 1.18 ns 0.09 0.19 False
BitwiseOrBenchmark - Duration of single invocation 13.15 ns 1.02 ns 0.08 0.13 False
MultiplyOperatorBenchmark - Duration of single invocation 38.69 ns 1.22 ns 0.03 0.08 False
DivisionOperatorBenchmark - Duration of single invocation 41.17 ns 0.58 ns 0.01 0.13 False
SubtractionOperatorBenchmark - Duration of single invocation 37.86 ns 1.04 ns 0.03 0.18 False
AddOperatorBenchmark - Duration of single invocation 38.00 ns 1.03 ns 0.03 0.13 False
InequalityOperatorBenchmark - Duration of single invocation 8.36 ns 0.05 ns 0.01 0.26 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.20 ns 0.98 ns 0.07 0.15 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.58 ns 1.07 ns 0.08 0.15 False
OnesComplementBenchmark - Duration of single invocation 11.10 ns 1.34 ns 0.12 0.21 False
SubtractBenchmark - Duration of single invocation 37.88 ns 1.04 ns 0.03 0.16 False
UnaryNegateOperatorBenchmark - Duration of single invocation 35.81 ns 1.02 ns 0.03 0.16 False
XorBenchmark - Duration of single invocation 13.18 ns 1.00 ns 0.08 0.18 False
MultiplyBenchmark - Duration of single invocation 38.75 ns 1.16 ns 0.03 0.10 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.20 ns 1.04 ns 0.08 0.18 False
BitwiseAndBenchmark - Duration of single invocation 13.19 ns 1.18 ns 0.09 0.16 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Float*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Float* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2553987281938601 < 35.94699929401828.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.551453607285936 (T) = (0 -1.1019892043155233) / Math.Sqrt((179.0994403541137 / (299)) + (0.022321570004502057 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9715312234348412 = (38.7086955350298 - 1.1019892043155233) / 38.7086955350298 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0751173390174458 < 6.24330207769909.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 186.10142058832383 (T) = (0 -0.9938472097682919) / Math.Sqrt((0.04957901962772271 / (299)) + (0.01319339109751031 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8358482971505653 = (6.054443496573907 - 0.9938472097682919) / 6.054443496573907 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9554740139391183 < 33.88726872878973.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.28122351979976 (T) = (0 -1.0731967038257837) / Math.Sqrt((96.34657557409669 / (299)) + (0.005968790842394488 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9630684410864184 = (29.05906859596748 - 1.0731967038257837) / 29.05906859596748 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.004679477290501698 < 7.868932290231129.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 76.34452736544517 (T) = (0 -0.03948237749275646) / Math.Sqrt((5.521172859857063 / (299)) + (0.005708091819374225 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9962337640985018 = (10.483246011502033 - 0.03948237749275646) / 10.483246011502033 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0434396570267166 < 39.16560788207255.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 191.17672493306483 (T) = (0 -1.0716137138464328) / Math.Sqrt((16.261350727341682 / (299)) + (0.013727189758420945 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9766530144698243 = (45.89944652432271 - 1.0716137138464328) / 45.89944652432271 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1815689867012296 < 10.661879781778948.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.8089895755453 (T) = (0 -1.0759505323855925) / Math.Sqrt((7.19355141332896 / (299)) + (0.006566015081085339 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8739720179838223 = (8.537393959442095 - 1.0759505323855925) / 8.537393959442095 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0175549759275748 < 12.625545915245837.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.91535879202539 (T) = (0 -1.0120551547202694) / Math.Sqrt((18.960829647622965 / (299)) + (0.028814061800399134 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9247603106334935 = (13.451080981878595 - 1.0120551547202694) / 13.451080981878595 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.216515302828329 < 36.76279153769067.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 335.7552398579071 (T) = (0 -1.1369666663804996) / Math.Sqrt((2.8270413263962766 / (299)) + (0.08998882788695285 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9715447816866815 = (39.95635014503971 - 1.1369666663804996) / 39.95635014503971 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.5831366775377174 < 39.05327577794914.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 408.62181474860853 (T) = (0 -1.0529071494979683) / Math.Sqrt((2.7778463634655575 / (299)) + (0.020634657465191376 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9751065921274888 = (42.29662547170384 - 1.0529071494979683) / 42.29662547170384 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0358057996655952 < 35.9558201319017.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.69405854045517 (T) = (0 -1.072170623140086) / Math.Sqrt((143.03815193020367 / (299)) + (0.008843014927508846 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9697615452668491 = (35.45718961507156 - 1.072170623140086) / 35.45718961507156 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0339649612458923 < 35.95598202036198.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.78722511389363 (T) = (0 -1.0620505908761126) / Math.Sqrt((142.2513784994868 / (299)) + (0.010724736892521625 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9700152471780417 = (35.41968803886074 - 1.0620505908761126) / 35.41968803886074 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.04991218052789881 < 7.937533162211661.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 76.00333104689254 (T) = (0 -0.07112671858025854) / Math.Sqrt((5.451452967385607 / (299)) + (0.01512174192149817 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9932369918135014 = (10.517023877370546 - 0.07112671858025854) / 10.517023877370546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9775614700399752 < 12.571351691254307.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.289268694510326 (T) = (0 -1.0934739287803101) / Math.Sqrt((16.03324451637232 / (299)) + (0.03335289846120833 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9136310802754166 = (12.660502554243164 - 1.0934739287803101) / 12.660502554243164 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0677742076042505 < 12.541536711824543.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.574481222967535 (T) = (0 -1.0531034841165074) / Math.Sqrt((16.032498331862037 / (299)) + (0.02629859187710284 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9167793936157335 = (12.654359657677333 - 1.0531034841165074) / 12.654359657677333 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3396173828195077 < 10.526483629386194.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.69181878384261 (T) = (0 -1.1240845620185729) / Math.Sqrt((7.131227986130819 / (299)) + (0.05184179828487715 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8680790967367491 = (8.520898009434024 - 1.1240845620185729) / 8.520898009434024 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.043198976399381 < 35.971685738369466.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.55954202573848 (T) = (0 -1.0819472410047763) / Math.Sqrt((179.37650315791538 / (299)) + (0.01034420678903683 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9720482244214196 = (38.70763908944222 - 1.0819472410047763) / 38.70763908944222 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0216149051370864 < 33.92605213106175.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.20821339400399 (T) = (0 -1.0925882002699665) / Math.Sqrt((96.46195940338056 / (299)) + (0.014510670280081002 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9624149951401174 = (29.0697900490674 - 1.0925882002699665) / 29.0697900490674 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9977840513917069 < 12.543089579390116.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.62226809703301 (T) = (0 -1.151721763561023) / Math.Sqrt((18.982375323840586 / (299)) + (0.07579532435691137 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9144231809378595 = (13.458338089485588 - 1.151721763561023) / 13.458338089485588 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1598742711964174 < 36.75810407828732.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 187.05317207978814 (T) = (0 -1.040545948353989) / Math.Sqrt((15.236404341866551 / (299)) + (0.009164261167090817 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.976041074855261 = (43.430410257051015 - 1.040545948353989) / 43.430410257051015 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0435366986024894 < 12.59087742201826.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.68645813518616 (T) = (0 -1.0866618614507615) / Math.Sqrt((16.00030902809044 / (299)) + (0.016480835304497756 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9141470072461245 = (12.657239154912379 - 1.0866618614507615) / 12.657239154912379 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Float.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1841313951272603 < 12.548883423768311.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.99454125845711 (T) = (0 -1.0765471986737836) / Math.Sqrt((18.913615105479224 / (299)) + (0.011058898623143406 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9199469803070086 = (13.447927421131805 - 1.0765471986737836) / 13.447927421131805 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AndNotBenchmark - Duration of single invocation 17.71 ns 1.09 ns 0.06 0.14 False
OnesComplementOperatorBenchmark - Duration of single invocation 8.99 ns 1.10 ns 0.12 0.17 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.30 ns 1.19 ns 0.13 0.19 False
BitwiseOrBenchmark - Duration of single invocation 9.52 ns 1.03 ns 0.11 0.17 False
OnesComplementBenchmark - Duration of single invocation 9.00 ns 1.09 ns 0.12 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.49 ns 1.05 ns 0.11 0.22 False
OneBenchmark - Duration of single invocation 5.82 ns 1.25 ns 0.21 0.35 False
ConditionalSelectBenchmark - Duration of single invocation 33.00 ns 3.66 ns 0.11 0.12 False
BitwiseAndBenchmark - Duration of single invocation 9.59 ns 1.02 ns 0.11 0.15 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.52 ns 1.05 ns 0.11 0.15 False
XorBenchmark - Duration of single invocation 9.53 ns 1.24 ns 0.13 0.14 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt64>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0912051895742432 < 16.829513953285637.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.18743069860824 (T) = (0 -1.2120533654078651) / Math.Sqrt((26.90262519112322 / (299)) + (0.019601237506665977 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9244165863106375 = (16.035970145371298 - 1.2120533654078651) / 16.035970145371298 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0981805592643041 < 8.557420597247173.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 40.92290576834518 (T) = (0 -1.2003914576454786) / Math.Sqrt((3.962049666360687 / (299)) + (0.027276847082735815 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8037754244928661 = (6.117436893636379 - 1.2003914576454786) / 6.117436893636379 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1860871606608565 < 9.11753036150609.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.62691055411747 (T) = (0 -1.2079753042122938) / Math.Sqrt((8.25185632605875 / (299)) + (0.07442728624839667 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8689817870244059 = (9.219903681920265 - 1.2079753042122938) / 9.219903681920265 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0255231418988466 < 9.089507086154692.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.649570754289215 (T) = (0 -1.2160121420858696) / Math.Sqrt((10.927960081484526 / (299)) + (0.03102326629590553 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8796350152513273 = (10.102706735060494 - 1.2160121420858696) / 10.102706735060494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0852284967708656 < 8.591523891025899.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 40.89235988656378 (T) = (0 -1.217819327263767) / Math.Sqrt((3.8848956874293346 / (299)) + (0.03371165103745261 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8014929338217908 = (6.134891571922546 - 1.217819327263767) / 6.134891571922546 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.052303730598432 < 9.05427795665917.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.39754973037204 (T) = (0 -1.2679494291793632) / Math.Sqrt((8.317176562976902 / (299)) + (0.06550206440366851 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8624433120627711 = (9.217650178942705 - 1.2679494291793632) / 9.217650178942705 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.250179400189516 < 5.609057707248157.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 29.56949079261517 (T) = (0 -1.2366376611262284) / Math.Sqrt((9.886049792511445 / (299)) + (0.047123442757527015 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8177344778023463 = (6.784813969288087 - 1.2366376611262284) / 6.784813969288087 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 3.663017415216087 < 31.331483208540657.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.11440789010736 (T) = (0 -1.3738163858790045) / Math.Sqrt((110.1350802833062 / (299)) + (0.2690608132310944 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9565957906515192 = (31.65168555079534 - 1.3738163858790045) / 31.65168555079534 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0175348078699966 < 9.105464282666762.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.4890013417799 (T) = (0 -1.2445190552913181) / Math.Sqrt((10.98138251656547 / (299)) + (0.03024074534589102 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8769853860170929 = (10.116839089250357 - 1.2445190552913181) / 10.116839089250357 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0523162226416607 < 9.062077201378779.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.363662233547295 (T) = (0 -1.2978750201302403) / Math.Sqrt((8.241322850189723 / (299)) + (0.2054437415532428 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8595937945996244 = (9.243715521185702 - 1.2978750201302403) / 9.243715521185702 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt64>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2427096537362998 < 9.06736022816907.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.086741674559775 (T) = (0 -1.309889053699943) / Math.Sqrt((10.838423531510028 / (299)) + (0.0828132900785852 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8704444015081113 = (10.110632569706764 - 1.309889053699943) / 10.110632569706764 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NegateBenchmark - Duration of single invocation 38.30 ns 1.44 ns 0.04 0.11 False
AddOperatorBenchmark - Duration of single invocation 38.79 ns 0.93 ns 0.02 0.11 False
OnesComplementBenchmark - Duration of single invocation 9.10 ns 1.24 ns 0.14 0.16 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.54 ns 1.18 ns 0.12 0.15 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.59 ns 0.96 ns 0.10 0.17 False
SubtractionOperatorBenchmark - Duration of single invocation 37.58 ns 1.24 ns 0.03 0.13 False
MultiplyBenchmark - Duration of single invocation 39.79 ns 0.99 ns 0.02 0.04 True
ConditionalSelectBenchmark - Duration of single invocation 32.74 ns 1.21 ns 0.04 0.40 False
AndNotBenchmark - Duration of single invocation 17.73 ns 1.30 ns 0.07 0.12 False
AddBenchmark - Duration of single invocation 38.11 ns 1.07 ns 0.03 0.12 False
MultiplyOperatorBenchmark - Duration of single invocation 40.19 ns 1.29 ns 0.03 0.05 True
UnaryNegateOperatorBenchmark - Duration of single invocation 38.36 ns 1.09 ns 0.03 0.08 False
SubtractBenchmark - Duration of single invocation 38.23 ns 1.24 ns 0.03 0.10 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.09 ns 1.27 ns 0.14 0.18 False
OnesComplementOperatorBenchmark - Duration of single invocation 9.34 ns 0.98 ns 0.10 0.16 False
OneBenchmark - Duration of single invocation 22.40 ns 1.31 ns 0.06 0.25 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<SByte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4429153349677033 < 36.23029518426999.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.443833576163975 (T) = (0 -1.2140430531486262) / Math.Sqrt((146.31682820329985 / (299)) + (0.03001687360037638 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9661335841423686 = (35.84799342960464 - 1.2140430531486262) / 35.84799342960464 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9317329445190736 < 36.56172711245007.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.68406759788131 (T) = (0 -1.2300117284850287) / Math.Sqrt((141.48851621709315 / (299)) + (0.038197261653216935 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9653201108492533 = (35.467579585921136 - 1.2300117284850287) / 35.467579585921136 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.242523889114185 < 8.65923064888196.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.90543551761194 (T) = (0 -1.2175031563432674) / Math.Sqrt((5.479448819091446 / (299)) + (0.03102211104236686 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8380168503531805 = (7.516233379816691 - 1.2175031563432674) / 7.516233379816691 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1804091246133157 < 9.100235915089637.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.96753493170831 (T) = (0 -1.223080224056925) / Math.Sqrt((16.168179170829507 / (299)) + (0.022766629293572607 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8918176153795911 = (11.30572438709387 - 1.223080224056925) / 11.30572438709387 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9582587470247541 < 9.077255029756085.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.0634685421938 (T) = (0 -1.2590151801936105) / Math.Sqrt((15.964277783380988 / (299)) + (0.027733714105401628 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8887925662966647 = (11.321322129887896 - 1.2590151801936105) / 11.321322129887896 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2435775841255334 < 35.92690465285832.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.77944354804727 (T) = (0 -1.2355764125920392) / Math.Sqrt((141.03950700623776 / (299)) + (0.01916169818708234 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9651504938700316 = (35.454631924597685 - 1.2355764125920392) / 35.454631924597685 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9939485472428701 < 37.93196582303564.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 373.35894807534186 (T) = (0 -1.2517837743751645) / Math.Sqrt((3.294225449710834 / (299)) + (0.030373734886136493 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9707002302052202 = (42.72333138256225 - 1.2517837743751645) / 42.72333138256225 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2109640570142877 < 31.046190222037907.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.0481104842266 (T) = (0 -1.3962911308654136) / Math.Sqrt((361.00020295452384 / (299)) + (0.19413633593909294 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9700734607785321 = (46.657287049876395 - 1.3962911308654136) / 46.657287049876395 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3015739559070874 < 16.828284653896354.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.091683517575454 (T) = (0 -1.224479396208972) / Math.Sqrt((51.88721510140133 / (299)) + (0.017687934224294002 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9389292662001902 = (20.050183124093785 - 1.224479396208972) / 20.050183124093785 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0669104003068153 < 36.25243466517998.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.634856792130414 (T) = (0 -1.2191005808732962) / Math.Sqrt((153.69754127340389 / (299)) + (0.02781272581966363 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.966914865190441 = (36.84738139622362 - 1.2191005808732962) / 36.84738139622362 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2881083825411064 < 37.99860271899436.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 277.36626609506277 (T) = (0 -1.4075657595490267) / Math.Sqrt((0.5509458767210761 / (299)) + (0.4236223350124703 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.965575246576339 = (40.88818712007309 - 1.4075657595490267) / 40.88818712007309 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0948518996240846 < 36.393181983131065.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.49045364561081 (T) = (0 -1.169496343850459) / Math.Sqrt((141.93223929747577 / (299)) + (0.017314460032070114 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9668656167269122 = (35.295551880705766 - 1.169496343850459) / 35.295551880705766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2367853725333424 < 36.43005217220565.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.37665715187596 (T) = (0 -1.2850003743080771) / Math.Sqrt((154.66137687117316 / (299)) + (0.028375630982888 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9651207533462441 = (36.84140277065602 - 1.2850003743080771) / 36.84140277065602 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2672629343586368 < 9.075084041308.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.94538850735213 (T) = (0 -1.2411826747413433) / Math.Sqrt((16.06995218926205 / (299)) + (0.023270905699502683 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8900687509269463 = (11.290535541141066 - 1.2411826747413433) / 11.290535541141066 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9769822644863946 < 8.607734880430934.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.02308617945957 (T) = (0 -1.2098042143944319) / Math.Sqrt((5.473365125796831 / (299)) + (0.033910030694211904 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8395889439557269 = (7.541900441454168 - 1.2098042143944319) / 7.541900441454168 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<SByte>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.311041168821004 < 19.553396124968454.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 32.1233719302394 (T) = (0 -1.2698299657458516) / Math.Sqrt((81.86503404592013 / (299)) + (0.01648760369538968 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9298457576556604 = (18.10054421959426 - 1.2698299657458516) / 18.10054421959426 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AddBenchmark - Duration of single invocation 15.59 ns 1.04 ns 0.07 0.15 False
OnesComplementBenchmark - Duration of single invocation 11.03 ns 1.00 ns 0.09 0.17 False
InequalityOperatorBenchmark - Duration of single invocation 3.06 ns 0.00 ns 0.00 0.40 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.28 ns 1.24 ns 0.09 0.12 False
BitwiseOrBenchmark - Duration of single invocation 13.02 ns 1.16 ns 0.09 0.13 False
MultiplyBenchmark - Duration of single invocation 15.74 ns 1.17 ns 0.07 0.06 True
BitwiseAndOperatorBenchmark - Duration of single invocation 13.20 ns 1.41 ns 0.11 0.13 False
OnesComplementOperatorBenchmark - Duration of single invocation 10.96 ns 1.01 ns 0.09 0.15 False
MultiplyOperatorBenchmark - Duration of single invocation 15.75 ns 1.23 ns 0.08 0.08 True
EqualityOperatorBenchmark - Duration of single invocation 3.10 ns 0.00 ns 0.00 0.44 False
UnaryNegateOperatorBenchmark - Duration of single invocation 14.59 ns 1.01 ns 0.07 0.12 False
NegateBenchmark - Duration of single invocation 14.53 ns 1.11 ns 0.08 0.12 False
SubtractionOperatorBenchmark - Duration of single invocation 15.73 ns 1.30 ns 0.08 0.13 False
BitwiseAndBenchmark - Duration of single invocation 13.19 ns 0.99 ns 0.08 0.13 False
AddOperatorBenchmark - Duration of single invocation 15.52 ns 1.17 ns 0.08 0.12 False
XorBenchmark - Duration of single invocation 13.15 ns 0.98 ns 0.07 0.10 False
BitwiseOrOperatorBenchmark - Duration of single invocation 14.70 ns 1.36 ns 0.09 0.13 False
SubtractBenchmark - Duration of single invocation 15.45 ns 1.18 ns 0.08 0.12 False
AllBitsSetBenchmark - Duration of single invocation 6.36 ns 1.05 ns 0.17 0.13 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.037485431582039 < 14.932557117758494.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.68125559744273 (T) = (0 -1.1720717349728702) / Math.Sqrt((33.933399822439654 / (299)) + (0.015447477833096109 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9321815828464346 = (17.282499122898678 - 1.1720717349728702) / 17.282499122898678 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9989775239072181 < 10.711825509915625.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.27267552885427 (T) = (0 -1.210665552577408) / Math.Sqrt((7.327723522972432 / (299)) + (0.06716631259040257 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8585757520583551 = (8.560523179002221 - 1.210665552577408) / 8.560523179002221 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 3.021844733421663.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 60.83968593536266 (T) = (0 -0.06666224080229284) / Math.Sqrt((2.093283594719773 / (299)) + (0.008350593266480884 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9873926038049665 = (5.287550242020096 - 0.06666224080229284) / 5.287550242020096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2428730462243507 < 12.701342546771484.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.393660017095755 (T) = (0 -1.2310893869269255) / Math.Sqrt((16.232306958271884 / (299)) + (0.09378336598403597 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9029183504593614 = (12.68096898592137 - 1.2310893869269255) / 12.68096898592137 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1630673865293735 < 12.70148927065443.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.32302573535854 (T) = (0 -1.1988861989919157) / Math.Sqrt((19.068669487742206 / (299)) + (0.017386152508271146 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9110244705305113 = (13.474336215139184 - 1.1988861989919157) / 13.474336215139184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1651865104415775 < 15.124020386964068.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 130.8022305339436 (T) = (0 -1.2609835149694981) / Math.Sqrt((3.6558968547621626 / (299)) + (0.14957762862775337 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9341951781930997 = (19.16247898474324 - 1.2609835149694981) / 19.16247898474324 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4136183097602557 < 12.750915878989998.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.17854068029556 (T) = (0 -1.2372711796595706) / Math.Sqrt((16.37781011064412 / (299)) + (0.09008051572536065 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9023227386852118 = (12.666931515126851 - 1.2372711796595706) / 12.666931515126851 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0055211170565128 < 10.596215768429373.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.43334871494899 (T) = (0 -1.1631253936852737) / Math.Sqrt((7.33803002711674 / (299)) + (0.011141269421047656 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8633542321936373 = (8.511975250733778 - 1.1631253936852737) / 8.511975250733778 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2342773274783634 < 15.138714124000952.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 249.8086553090519 (T) = (0 -1.1687072368235714) / Math.Sqrt((1.172233191969301 / (299)) + (0.013428465866491187 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.934834672813637 = (17.934495034165096 - 1.1687072368235714) / 17.934495034165096 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 2.885623143336114.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 59.84050730512657 (T) = (0 -0.050797051541904226) / Math.Sqrt((2.1168354823436015 / (299)) + (0.013483050773450389 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9903978101436021 = (5.290152798640817 - 0.050797051541904226) / 5.290152798640817 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0123220585159827 < 13.97707113161734.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.05891673179907 (T) = (0 -1.1639481161524134) / Math.Sqrt((18.861949770823284 / (299)) + (0.008710717294025151 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9139264977319642 = (13.52272285293841 - 1.1639481161524134) / 13.52272285293841 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1078644459934524 < 14.031010800986857.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.06550974715126 (T) = (0 -1.1589866042304318) / Math.Sqrt((18.888296949745673 / (299)) + (0.007946792709973957 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9143063585503176 = (13.524767819686655 - 1.1589866042304318) / 13.524767819686655 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2959668357466694 < 15.050131143774212.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.717819671331156 (T) = (0 -1.1927213977180653) / Math.Sqrt((27.46732872378662 / (299)) + (0.011167216443936088 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9254437741922648 = (15.997609653603469 - 1.1927213977180653) / 15.997609653603469 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9930072070123077 < 12.657442607333888.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.47220313443469 (T) = (0 -1.1828479564478744) / Math.Sqrt((19.06375719533953 / (299)) + (0.0731576063834377 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9121548093498963 = (13.465141889887612 - 1.1828479564478744) / 13.465141889887612 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1733113508582098 < 15.04530001925383.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.36755605738163 (T) = (0 -1.1446487906464617) / Math.Sqrt((26.874834292883243 / (299)) + (0.017531229199782332 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9284945755125312 = (16.007859527455288 - 1.1446487906464617) / 16.007859527455288 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9752100936112758 < 12.705087523578332.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.64040827854123 (T) = (0 -1.1202992703764239) / Math.Sqrt((19.095738884363342 / (299)) + (0.016764200513952447 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.916906852236884 = (13.482450725903426 - 1.1202992703764239) / 13.482450725903426 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3562742603519884 < 12.693769017254528.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.41314315637758 (T) = (0 -1.1713938302356) / Math.Sqrt((15.95815982859023 / (299)) + (0.021964927023612538 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9076818359349595 = (12.68866037468339 - 1.1713938302356) / 12.68866037468339 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1844027932970909 < 15.166670612079407.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.83823995583112 (T) = (0 -1.2899182024227316) / Math.Sqrt((33.78182145023547 / (299)) + (0.09293817232863918 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9254928128853775 = (17.312668111310014 - 1.2899182024227316) / 17.312668111310014 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int32>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0536561121203296 < 6.418370213261517.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 199.97897193601128 (T) = (0 -1.1710339590779433) / Math.Sqrt((0.07303476062217805 / (299)) + (0.008099157118086378 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8065935914470915 = (6.054783643622614 - 1.1710339590779433) / 6.054783643622614 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementOperatorBenchmark - Duration of single invocation 8.94 ns 1.16 ns 0.13 0.15 False
NegateBenchmark - Duration of single invocation 22.72 ns 1.12 ns 0.05 0.10 False
AddBenchmark - Duration of single invocation 22.88 ns 1.04 ns 0.05 0.12 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.54 ns 1.06 ns 0.11 0.16 False
OneBenchmark - Duration of single invocation 12.26 ns 1.32 ns 0.11 0.21 False
ConditionalSelectBenchmark - Duration of single invocation 32.61 ns 1.03 ns 0.03 0.41 False
MultiplyOperatorBenchmark - Duration of single invocation 23.65 ns 1.00 ns 0.04 0.04 True
BitwiseOrOperatorBenchmark - Duration of single invocation 9.34 ns 1.02 ns 0.11 0.13 False
UnaryNegateOperatorBenchmark - Duration of single invocation 22.56 ns 1.01 ns 0.04 0.11 False
OnesComplementBenchmark - Duration of single invocation 9.13 ns 1.11 ns 0.12 0.14 False
AndNotBenchmark - Duration of single invocation 17.68 ns 1.02 ns 0.06 0.10 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.49 ns 1.09 ns 0.12 0.18 False
AddOperatorBenchmark - Duration of single invocation 22.65 ns 0.98 ns 0.04 0.11 False
SubtractBenchmark - Duration of single invocation 22.85 ns 1.05 ns 0.05 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 22.63 ns 1.06 ns 0.05 0.11 False
MultiplyBenchmark - Duration of single invocation 23.48 ns 1.04 ns 0.04 0.05 True

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1630577870463885 < 8.485673789503938.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.42256743285315 (T) = (0 -1.1732270505823414) / Math.Sqrt((5.636419408831696 / (299)) + (0.013718553337558446 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.843727588480322 = (7.507576284087787 - 1.1732270505823414) / 7.507576284087787 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1174388910912574 < 21.60939984426568.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.21667185965124 (T) = (0 -1.1197780350767155) / Math.Sqrt((53.41484631448334 / (299)) + (0.011159439557025627 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9489853570365201 = (21.950129806423192 - 1.1197780350767155) / 21.950129806423192 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0368236602864769 < 21.70700928168459.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.75750346472648 (T) = (0 -1.2463844717660992) / Math.Sqrt((57.171185201345885 / (299)) + (0.08481346473913066 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9452652266609997 = (22.77134617963256 - 1.2463844717660992) / 22.77134617963256 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.055885456168843 < 9.08849538144432.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.48714843376722 (T) = (0 -1.1333434482722993) / Math.Sqrt((16.04027528342284 / (299)) + (0.027358800099412325 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8998526533638072 = (11.316759618099693 - 1.1333434482722993) / 11.316759618099693 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3160600377528808 < 11.681190633071592.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 31.84701396157189 (T) = (0 -1.2510088071149368) / Math.Sqrt((33.135859637429846 / (299)) + (0.012503526312277973 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8946859875107689 = (11.878844776167451 - 1.2510088071149368) / 11.878844776167451 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0347371440794935 < 31.157730103831888.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.305942261540075 (T) = (0 -1.1700439618581928) / Math.Sqrt((361.4676101372342 / (299)) + (0.20246212699960767 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9749730604526473 = (46.75138003367871 - 1.1700439618581928) / 46.75138003367871 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0015803705497628 < 22.45565614005946.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 245.42560294466577 (T) = (0 -1.2030304543173789) / Math.Sqrt((0.37952987133574945 / (299)) + (0.18361294443843582 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9515112398014027 = (24.81050143146744 - 1.2030304543173789) / 24.81050143146744 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0213973521550852 < 9.074988584395877.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.18382978094054 (T) = (0 -1.1642505976169948) / Math.Sqrt((16.21731208078377 / (299)) + (0.08495522360870013 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8971284985801773 = (11.317523138557512 - 1.1642505976169948) / 11.317523138557512 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.008378297284898 < 21.562851612372388.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.735386844070376 (T) = (0 -1.2397627449511939) / Math.Sqrt((50.14911059733527 / (299)) + (0.06188202089668953 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9419540230229851 = (21.358288886103438 - 1.2397627449511939) / 21.358288886103438 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1059755095117767 < 8.67260070368684.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.97964799649685 (T) = (0 -1.135406101318273) / Math.Sqrt((5.6364495990142265 / (299)) + (0.009418933411879158 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8489461759396301 = (7.516566418500594 - 1.135406101318273) / 7.516566418500594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0241615056717686 < 16.76171213917877.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.60895960710188 (T) = (0 -1.0919340436795613) / Math.Sqrt((51.49831567296663 / (299)) + (0.0038445027585623627 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9454834200246499 = (20.029393703223562 - 1.0919340436795613) / 20.029393703223562 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0930926596067414 < 9.01884078745355.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.312614235468835 (T) = (0 -1.1825847896439405) / Math.Sqrt((16.06594836463777 / (299)) + (0.022077580043063573 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8954932176228501 = (11.315866422680221 - 1.1825847896439405) / 11.315866422680221 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9796811902246699 < 21.68924898402092.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.75302217529948 (T) = (0 -1.1270670898870194) / Math.Sqrt((50.348084303137114 / (299)) + (0.014920690005470845 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9477788457486196 = (21.58257713840608 - 1.1270670898870194) / 21.58257713840608 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0511454783162795 < 21.881266410082983.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.45689464303878 (T) = (0 -1.141056185491) / Math.Sqrt((57.165468896162444 / (299)) + (0.014733281697582607 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.949958842190448 = (22.802353811110088 - 1.141056185491) / 22.802353811110088 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0564329353217254 < 21.695844840644835.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.264798530697725 (T) = (0 -1.3095593536411085) / Math.Sqrt((50.21423421692578 / (299)) + (0.19238381663637502 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9393029029726188 = (21.575321024831716 - 1.3095593536411085) / 21.575321024831716 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int16>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0396300766283524 < 22.39870457644168.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 201.54867713448846 (T) = (0 -1.2065866683074085) / Math.Sqrt((2.0516349424615097 / (299)) + (0.19286216118191896 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9537591750265844 = (26.09353680435178 - 1.2065866683074085) / 26.09353680435178 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 13.78 ns 1.06 ns 0.08 0.13 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.17 ns 1.45 ns 0.11 0.10 False
InequalityOperatorBenchmark - Duration of single invocation 3.01 ns 0.00 ns 0.00 0.36 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.26 ns 0.97 ns 0.07 0.12 False
NegateBenchmark - Duration of single invocation 12.47 ns 2.20 ns 0.18 0.14 False
SubtractBenchmark - Duration of single invocation 13.82 ns 1.00 ns 0.07 0.14 False
EqualityOperatorBenchmark - Duration of single invocation 3.27 ns 0.02 ns 0.01 0.42 False
OnesComplementBenchmark - Duration of single invocation 11.11 ns 1.11 ns 0.10 0.16 False
AddOperatorBenchmark - Duration of single invocation 13.82 ns 1.00 ns 0.07 0.14 False
BitwiseOrBenchmark - Duration of single invocation 13.18 ns 1.11 ns 0.08 0.13 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.39 ns 1.07 ns 0.08 0.12 False
XorBenchmark - Duration of single invocation 13.17 ns 1.08 ns 0.08 0.10 False
BitwiseAndBenchmark - Duration of single invocation 13.14 ns 0.97 ns 0.07 0.12 False
UnaryNegateOperatorBenchmark - Duration of single invocation 12.66 ns 1.18 ns 0.09 0.15 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.04 ns 1.70 ns 0.15 0.16 False
AllBitsSetBenchmark - Duration of single invocation 6.50 ns 0.99 ns 0.15 0.16 True
AddBenchmark - Duration of single invocation 13.76 ns 0.98 ns 0.07 0.15 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0570346959002355 < 13.177199489133184.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.89245628170256 (T) = (0 -1.1747091331656503) / Math.Sqrt((19.64862288855335 / (299)) + (0.010220200567818414 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9145697881651788 = (13.750511767861969 - 1.1747091331656503) / 13.750511767861969 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4548531629976076 < 12.871941536537106.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.974121793515856 (T) = (0 -1.1880716957924482) / Math.Sqrt((16.31181110348059 / (299)) + (0.007166775210457057 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9061520437639596 = (12.659537228538964 - 1.1880716957924482) / 12.659537228538964 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 2.889915593302027.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 56.40719210326165 (T) = (0 -0.11049861889627606) / Math.Sqrt((2.07440086144547 / (299)) + (0.035181687638417024 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9791551879018486 = (5.3010129511350215 - 0.11049861889627606) / 5.3010129511350215 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9661697947686755 < 12.694167182026225.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.868682474050445 (T) = (0 -1.1816689718969458) / Math.Sqrt((16.274211288813365 / (299)) + (0.02005541953999601 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9067620087270093 = (12.673685434053878 - 1.1816689718969458) / 12.673685434053878 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 2.1986734967284662 < 12.184347890064899.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.26944865545661 (T) = (0 -1.2194751606805674) / Math.Sqrt((12.110893897542038 / (299)) + (0.05739626996291768 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8893548221120997 = (11.021493968007105 - 1.2194751606805674) / 11.021493968007105 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9979743018825743 < 13.229944555840806.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.44556785901855 (T) = (0 -1.1263461069042087) / Math.Sqrt((25.44474750639283 / (299)) + (0.016842460960863673 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9250426964182282 = (15.026502463171783 - 1.1263461069042087) / 15.026502463171783 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.018071721948485615 < 2.8229175452695436.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 59.673105180452744 (T) = (0 -0.06071774489578087) / Math.Sqrt((2.1200679483573 / (299)) + (0.014272359318869766 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9885454915763061 = (5.300772643388631 - 0.06071774489578087) / 5.300772643388631 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1126302315228693 < 10.775826227481627.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.84420614294881 (T) = (0 -1.205894147863364) / Math.Sqrt((7.306626693911008 / (299)) + (0.05333512628381695 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8588136142841444 = (8.541150350645594 - 1.205894147863364) / 8.541150350645594 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.001111788681402 < 13.165043415969047.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.79851181766138 (T) = (0 -1.1475538343983545) / Math.Sqrt((19.718692286449993 / (299)) + (0.01312457710984265 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9164403857596279 = (13.733354860845093 - 1.1475538343983545) / 13.733354860845093 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1122187884542862 < 12.707640517841739.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.62931135603725 (T) = (0 -1.126899531521776) / Math.Sqrt((19.131960883671766 / (299)) + (0.009403098786161329 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9163225750953764 = (13.467187031702132 - 1.126899531521776) / 13.467187031702132 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.065925825013285 < 12.732532566935555.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.97315478213417 (T) = (0 -1.1845084078833157) / Math.Sqrt((16.1976210437971 / (299)) + (0.07018291465742688 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9064401326967529 = (12.660432747772896 - 1.1845084078833157) / 12.660432747772896 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0845733839507652 < 12.691703922986125.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.79571722812393 (T) = (0 -1.212961721834003) / Math.Sqrt((19.085420110351052 / (299)) + (0.1087783986883945 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9099238944370036 = (13.465965410612634 - 1.212961721834003) / 13.465965410612634 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9705404662384561 < 12.66887908673279.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.23551057541538 (T) = (0 -1.142183542904918) / Math.Sqrt((19.15903790845171 / (299)) + (0.029508450576638742 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9152298259104013 = (13.473884596458124 - 1.142183542904918) / 13.473884596458124 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.182690899529573 < 12.216468245549155.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.549396421597635 (T) = (0 -1.1382850844915293) / Math.Sqrt((12.30828007116407 / (299)) + (0.011296103391194811 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8969608707988729 = (11.047114754528407 - 1.1382850844915293) / 11.047114754528407 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.7035074340945147 < 10.795322894889635.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.99392716007922 (T) = (0 -1.1573396946286694) / Math.Sqrt((7.437447414082756 / (299)) + (0.020592263334761284 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8644915863807356 = (8.54072203870991 - 1.1573396946286694) / 8.54072203870991 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9918121723050183 < 6.256589893814055.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 171.22728353026022 (T) = (0 -1.1495067311663585) / Math.Sqrt((0.06941873700427038 / (299)) + (0.012950933297617807 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8101634968212149 = (6.05524602443699 - 1.1495067311663585) / 6.05524602443699 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt64>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9777300680387262 < 13.216615698500169.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.168712179466105 (T) = (0 -1.1477834510105027) / Math.Sqrt((25.21550957183444 / (299)) + (0.05627576975308559 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9236994203266647 = (15.04291914851097 - 1.1477834510105027) / 15.04291914851097 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Int

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 14.58 ns 1.10 ns 0.08 0.17 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.09 ns 1.08 ns 0.10 0.17 False
NegateBenchmark - Duration of single invocation 14.50 ns 1.34 ns 0.09 0.16 False
XorBenchmark - Duration of single invocation 13.15 ns 1.29 ns 0.10 0.14 False
MultiplyOperatorBenchmark - Duration of single invocation 15.70 ns 1.12 ns 0.07 0.15 False
OnesComplementBenchmark - Duration of single invocation 11.08 ns 1.12 ns 0.10 0.20 False
MultiplyBenchmark - Duration of single invocation 15.79 ns 1.06 ns 0.07 0.12 False
AddBenchmark - Duration of single invocation 15.64 ns 1.49 ns 0.10 0.19 False
BitwiseOrOperatorBenchmark - Duration of single invocation 13.21 ns 1.21 ns 0.09 0.19 False
SubtractionOperatorBenchmark - Duration of single invocation 15.64 ns 1.06 ns 0.07 0.15 False
BitwiseOrBenchmark - Duration of single invocation 13.17 ns 1.23 ns 0.09 0.11 False
BitwiseAndOperatorBenchmark - Duration of single invocation 13.17 ns 1.09 ns 0.08 0.15 False
BitwiseAndBenchmark - Duration of single invocation 13.18 ns 1.44 ns 0.11 0.16 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.60 ns 0.97 ns 0.07 0.16 False
SubtractBenchmark - Duration of single invocation 15.62 ns 1.27 ns 0.08 0.12 False
AllBitsSetBenchmark - Duration of single invocation 6.51 ns 1.03 ns 0.16 0.10 False
AddOperatorBenchmark - Duration of single invocation 15.60 ns 1.07 ns 0.07 0.18 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Int*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Int* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.098239945902561 < 13.790694516163809.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.4134137822502 (T) = (0 -1.1094934255171505) / Math.Sqrt((18.67717781369693 / (299)) + (0.016998770002741248 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9180113496451009 = (13.532280635363025 - 1.1094934255171505) / 13.532280635363025 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0820237100346686 < 10.541583817521875.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.872983514981755 (T) = (0 -1.0656677931577372) / Math.Sqrt((7.222572762052699 / (299)) + (0.004536071909317293 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8751627748235664 = (8.536458509484001 - 1.0656677931577372) / 8.536458509484001 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3394940819361205 < 13.78219832647317.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.60773712095399 (T) = (0 -1.0897368592791794) / Math.Sqrt((18.644192295045336 / (299)) + (0.007515115686337092 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9193367423927812 = (13.509705553742176 - 1.0897368592791794) / 13.509705553742176 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2885275180134055 < 12.529088039641618.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.71417662265204 (T) = (0 -1.1050044949486124) / Math.Sqrt((18.888423473681197 / (299)) + (0.026680387514367167 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9179091339765498 = (13.46074841789287 - 1.1050044949486124) / 13.46074841789287 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1224207088086053 < 14.965680150404651.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 228.5950609999487 (T) = (0 -1.0723260257927654) / Math.Sqrt((1.2230387763543098 / (299)) + (0.031016388481556675 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9401973120698944 = (17.931067363494545 - 1.0723260257927654) / 17.931067363494545 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.123417695989281 < 10.564398799667762.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.785881543866594 (T) = (0 -1.0653119896288978) / Math.Sqrt((7.128899328555934 / (299)) + (0.01346437817662431 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8751680450474195 = (8.533968646358009 - 1.0653119896288978) / 8.533968646358009 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.059598909188585 < 14.984526931688531.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 155.81740743785565 (T) = (0 -1.076178857758521) / Math.Sqrt((3.827306802676462 / (299)) + (0.015149165461679064 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9438115434436577 = (19.15302401445028 - 1.076178857758521) / 19.15302401445028 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4864645442823003 < 14.839260701183406.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.94372366705103 (T) = (0 -1.0828475730086817) / Math.Sqrt((33.735282947498035 / (299)) + (0.026479513874154956 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.937295192835753 = (17.26897222046 - 1.0828475730086817) / 17.26897222046 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2110560583393333 < 12.547643076362585.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.62179507193344 (T) = (0 -1.1050623020946573) / Math.Sqrt((15.976744435636366 / (299)) + (0.022572671561091967 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9128526351596278 = (12.680386883972899 - 1.1050623020946573) / 12.680386883972899 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0623850989384982 < 14.80093492488216.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.859909115826824 (T) = (0 -1.1091802366480719) / Math.Sqrt((27.015754064866364 / (299)) + (0.06244284651078411 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9307409244906246 = (16.014944301385096 - 1.1091802366480719) / 16.014944301385096 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2349853354044986 < 12.58147359646215.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.203809971717554 (T) = (0 -1.0655329887248635) / Math.Sqrt((18.864265116007143 / (299)) + (0.013199350625969263 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9209580917610899 = (13.48060810354186 - 1.0655329887248635) / 13.48060810354186 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.088578509039951 < 12.533771067269697.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 50.27235620675063 (T) = (0 -1.0737154002499203) / Math.Sqrt((15.78280273695835 / (299)) + (0.009783016949519567 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9152570148455391 = (12.67025699286922 - 1.0737154002499203) / 12.67025699286922 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4376010851386947 < 12.511225999492584.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.378572191643016 (T) = (0 -1.1232782825305194) / Math.Sqrt((19.19017452238214 / (299)) + (0.08636998775908816 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9166232452201984 = (13.472319539146161 - 1.1232782825305194) / 13.472319539146161 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9749256800135011 < 12.566794718603818.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 50.043761274835795 (T) = (0 -1.0816784300395637) / Math.Sqrt((15.816066150361157 / (299)) + (0.015441029305786697 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9145875399956547 = (12.664176046264604 - 1.0816784300395637) / 12.664176046264604 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2659908642747946 < 14.791715219302583.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.205193441196926 (T) = (0 -1.079728464339306) / Math.Sqrt((33.56852688391347 / (299)) + (0.011111140229022485 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9374663896809032 = (17.26637017804763 - 1.079728464339306) / 17.26637017804763 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.026370005806942 < 6.172567578765395.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 146.4047342040854 (T) = (0 -1.079380885982193) / Math.Sqrt((0.04419297164974591 / (299)) + (0.022151991836221668 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8217186079667459 = (6.054366491489254 - 1.079380885982193) / 6.054366491489254 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Int.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0655656741491393 < 14.820070766713506.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.53270989672432 (T) = (0 -1.0715357440179396) / Math.Sqrt((27.033073681003025 / (299)) + (0.009701692736212058 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9330289892247231 = (15.99999360340414 - 1.0715357440179396) / 15.99999360340414 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
UnaryNegateOperatorBenchmark - Duration of single invocation 38.31 ns 1.07 ns 0.03 0.10 False
NegateBenchmark - Duration of single invocation 38.38 ns 1.09 ns 0.03 0.11 False
AndNotBenchmark - Duration of single invocation 17.69 ns 1.10 ns 0.06 0.12 False
MultiplyOperatorBenchmark - Duration of single invocation 39.46 ns 1.27 ns 0.03 0.06 True
OneBenchmark - Duration of single invocation 20.75 ns 1.30 ns 0.06 0.22 False
OnesComplementOperatorBenchmark - Duration of single invocation 9.13 ns 1.10 ns 0.12 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.46 ns 1.02 ns 0.11 0.16 False
ConditionalSelectBenchmark - Duration of single invocation 32.91 ns 1.14 ns 0.03 0.39 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.49 ns 1.25 ns 0.13 0.13 False
MultiplyBenchmark - Duration of single invocation 39.84 ns 1.29 ns 0.03 0.07 True
AddBenchmark - Duration of single invocation 38.26 ns 1.00 ns 0.03 0.12 False
SubtractBenchmark - Duration of single invocation 37.52 ns 1.26 ns 0.03 0.12 False
AddOperatorBenchmark - Duration of single invocation 38.72 ns 1.30 ns 0.03 0.11 False
OnesComplementBenchmark - Duration of single invocation 8.60 ns 1.06 ns 0.12 0.19 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.49 ns 1.31 ns 0.14 0.15 False
SubtractionOperatorBenchmark - Duration of single invocation 38.77 ns 1.17 ns 0.03 0.11 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Byte>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.067288866423577 < 36.27047886972653.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.386783890662734 (T) = (0 -1.207735939443431) / Math.Sqrt((141.26690524692688 / (299)) + (0.07840213566427236 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9657637300799987 = (35.27650477886487 - 1.207735939443431) / 35.27650477886487 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.091042324076078 < 36.3506773123944.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.28111435403784 (T) = (0 -1.2819849512876538) / Math.Sqrt((146.68917782053526 / (299)) + (0.055063559159523705 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9642741625180923 = (35.8839719834945 - 1.2819849512876538) / 35.8839719834945 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1010612779372972 < 16.812694373551818.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.89078959795058 (T) = (0 -1.3380165619270235) / Math.Sqrt((51.86886482655031 / (299)) + (0.18323194054756237 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9332116883223772 = (20.033693446024365 - 1.3380165619270235) / 20.033693446024365 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2676651507860668 < 37.63266990201574.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 720.7145353794452 (T) = (0 -1.3082871394748725) / Math.Sqrt((0.5201900736382169 / (299)) + (0.02970034673588136 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9680807784678114 = (40.987438811925394 - 1.3082871394748725) / 40.987438811925394 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2985982330218249 < 19.62210183171891.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 32.154437965041566 (T) = (0 -1.2540587360700577) / Math.Sqrt((81.72939990221401 / (299)) + (0.017317470598501087 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9306699123942589 = (18.088232387668445 - 1.2540587360700577) / 18.088232387668445 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.104227573292958 < 8.66522850051008.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.63033884437839 (T) = (0 -1.183443983006654) / Math.Sqrt((5.538645025273503 / (299)) + (0.017759816670422698 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.842668273560026 = (7.521966546640343 - 1.183443983006654) / 7.521966546640343 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.018252124931857 < 9.140378522204458.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.38245715109771 (T) = (0 -1.1900386051157192) / Math.Sqrt((15.932458526207292 / (299)) + (0.029583159609920057 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.8949647156740118 = (11.329893690031891 - 1.1900386051157192) / 11.329893690031891 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1413553438817459 < 31.297686144661824.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.36039340532526 (T) = (0 -1.262745014388979) / Math.Sqrt((359.30258451208476 / (299)) + (0.02620122262870185 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9729164120166125 = (46.623992920122596 - 1.262745014388979) / 46.623992920122596 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2487838462501253 < 9.0734637137535.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 40.36970564963259 (T) = (0 -1.3070613217523317) / Math.Sqrt((15.971118628230816 / (299)) + (0.18686952116747227 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8845522273038386 = (11.32166772235869 - 1.3070613217523317) / 11.32166772235869 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2945866788930007 < 37.952103831916986.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 372.0613028591057 (T) = (0 -1.2345089261798623) / Math.Sqrt((3.2822965732421614 / (299)) + (0.033778557295750945 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9711176221829672 = (42.742634765058476 - 1.2345089261798623) / 42.742634765058476 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0046203343403097 < 36.31923852310385.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.586813712513674 (T) = (0 -1.2325537724910405) / Math.Sqrt((153.97994560980473 / (299)) + (0.021685071486785893 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9665519206442582 = (36.84976226533181 - 1.2325537724910405) / 36.84976226533181 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2552997868790465 < 36.61138267899033.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.281585499570426 (T) = (0 -1.3059692591619148) / Math.Sqrt((154.07047826654892 / (299)) + (0.11571614544699091 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9645639509188837 = (36.85425697916915 - 1.3059692591619148) / 36.85425697916915 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3035926866773684 < 36.483418784913404.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.76736298017029 (T) = (0 -1.2527004633455856) / Math.Sqrt((140.65761492883055 / (299)) + (0.020005077782898974 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.96463147979072 = (35.41851499393232 - 1.2527004633455856) / 35.41851499393232 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0606536614027071 < 8.671855437943805.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.27393150265897 (T) = (0 -1.1908856052206338) / Math.Sqrt((5.555661224429916 / (299)) + (0.08686201492483897 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8414657675957433 = (7.511851460471439 - 1.1908856052206338) / 7.511851460471439 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3110858276217396 < 9.084085667404109.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.6950465694097 (T) = (0 -1.2937858039281462) / Math.Sqrt((16.077073111069488 / (299)) + (0.026621864706836064 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8855057812442119 = (11.30000988685502 - 1.2937858039281462) / 11.30000988685502 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Byte>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1715036105472267 < 36.755979401159365.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.78393782287246 (T) = (0 -1.2640224148685533) / Math.Sqrt((140.56924334376873 / (299)) + (0.014157801552459041 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9643145990399838 = (35.421275391716435 - 1.2640224148685533) / 35.421275391716435 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementOperatorBenchmark - Duration of single invocation 9.11 ns 1.30 ns 0.14 0.15 False
SubtractBenchmark - Duration of single invocation 10.70 ns 0.97 ns 0.09 0.14 False
BitwiseAndOperatorBenchmark - Duration of single invocation 9.53 ns 1.26 ns 0.13 0.15 False
UnaryNegateOperatorBenchmark - Duration of single invocation 11.09 ns 1.10 ns 0.10 0.14 False
AddOperatorBenchmark - Duration of single invocation 10.63 ns 1.35 ns 0.13 0.10 False
SubtractionOperatorBenchmark - Duration of single invocation 10.62 ns 1.25 ns 0.12 0.16 False
NegateBenchmark - Duration of single invocation 10.73 ns 1.29 ns 0.12 0.14 False
ConditionalSelectBenchmark - Duration of single invocation 33.16 ns 0.92 ns 0.03 0.34 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.67 ns 1.52 ns 0.16 0.18 False
AndNotBenchmark - Duration of single invocation 17.75 ns 1.26 ns 0.07 0.13 False
OnesComplementBenchmark - Duration of single invocation 9.14 ns 1.18 ns 0.13 0.15 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.51 ns 1.00 ns 0.10 0.16 False
MultiplyBenchmark - Duration of single invocation 10.98 ns 1.26 ns 0.11 0.09 True
MultiplyOperatorBenchmark - Duration of single invocation 11.03 ns 1.01 ns 0.09 0.08 True
OneBenchmark - Duration of single invocation 8.81 ns 1.22 ns 0.14 0.37 False
AddBenchmark - Duration of single invocation 10.62 ns 1.41 ns 0.13 0.12 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3035484348598816 < 8.64530261626004.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.1313992368993 (T) = (0 -1.2599324025218182) / Math.Sqrt((5.530911102122433 / (299)) + (0.017810209270442957 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8325737973300608 = (7.525299997430055 - 1.2599324025218182) / 7.525299997430055 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9734948317147375 < 10.14533678458027.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.48364958260072 (T) = (0 -1.2679446478614762) / Math.Sqrt((13.94280554904296 / (299)) + (0.03385352832563786 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8914643244837004 = (11.68228457444907 - 1.2679446478614762) / 11.68228457444907 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2593534631155838 < 9.065473948456152.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.23779221782179 (T) = (0 -1.2649036805897163) / Math.Sqrt((15.872678983761224 / (299)) + (0.026583697516477812 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8884064668778153 = (11.334919194688126 - 1.2649036805897163) / 11.334919194688126 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0979390256004329 < 10.400363533670077.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.78341473938551 (T) = (0 -1.2533515156141262) / Math.Sqrt((10.65545040062328 / (299)) + (0.01872848890353697 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8792097083794255 = (10.376260366612435 - 1.2533515156141262) / 10.376260366612435 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.350774126842148 < 10.251084529127573.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.21386453946231 (T) = (0 -1.1852361935536997) / Math.Sqrt((10.948551964052001 / (299)) + (0.017606224959160834 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8892359093500706 = (10.70054551614246 - 1.1852361935536997) / 10.70054551614246 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.254356289446951 < 10.212715817474301.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.95054009586892 (T) = (0 -1.2624139736757303) / Math.Sqrt((10.907660660062765 / (299)) + (0.01490816742951176 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8819577456316706 = (10.694593901405822 - 1.2624139736757303) / 10.694593901405822 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2906192588634646 < 10.199875105858093.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.066868589234964 (T) = (0 -1.1770648516331286) / Math.Sqrt((12.145457425897542 / (299)) + (0.011489991947361578 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8922503948067901 = (10.92407577292268 - 1.1770648516331286) / 10.92407577292268 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9231329527982977 < 31.08837401644105.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 40.90589850288271 (T) = (0 -1.3848730909151903) / Math.Sqrt((366.5470219657404 / (299)) + (0.3418851921464495 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9705032193993477 = (46.94997429260359 - 1.3848730909151903) / 46.94997429260359 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.5238787663355382 < 9.128564719234921.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.71474467678351 (T) = (0 -1.2666676503661132) / Math.Sqrt((15.94654464533339 / (299)) + (0.047114818841200605 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8880920594421983 = (11.318836215307387 - 1.2666676503661132) / 11.318836215307387 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2641439213188592 < 16.876489203384804.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.326770912492314 (T) = (0 -1.2669554188837415) / Math.Sqrt((51.171374208213244 / (299)) + (0.012420889467200913 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9368035062513008 = (20.047875186268897 - 1.2669554188837415) / 20.047875186268897 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1780015389835043 < 8.672134293945012.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.69108645232053 (T) = (0 -1.1478807352650562) / Math.Sqrt((5.575825563574982 / (299)) + (0.017341724356969953 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8471994021798499 = (7.5122790855578865 - 1.1478807352650562) / 7.5122790855578865 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9954871464885958 < 9.092387838953137.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.07730340325246 (T) = (0 -1.3402647645618506) / Math.Sqrt((16.042542005776944 / (299)) + (0.12048328879503015 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8814848344061832 = (11.308803880469586 - 1.3402647645618506) / 11.308803880469586 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2626606140770624 < 10.449160192951704.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 108.13662878333892 (T) = (0 -1.3666015708418233) / Math.Sqrt((0.9969342327840981 / (299)) + (0.19241598290676185 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.895387284916382 = (13.063436598022374 - 1.3666015708418233) / 13.063436598022374 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.00792851287508 < 10.488522384498525.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 97.18960434482285 (T) = (0 -1.4200724752743359) / Math.Sqrt((0.19088299357228364 / (299)) + (0.2601135046123151 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8820873589243421 = (12.043428612231283 - 1.4200724752743359) / 12.043428612231283 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2190789239879096 < 8.346101846083275.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 31.25930159966048 (T) = (0 -1.1495735367021183) / Math.Sqrt((16.834955076156685 / (299)) + (0.013684020145887621 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8664222362997654 = (8.606024721913354 - 1.1495735367021183) / 8.606024721913354 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.4081336426422844 < 10.184874980169138.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 47.34127437728857 (T) = (0 -1.2980589588067477) / Math.Sqrt((14.052318545137224 / (299)) + (0.02317195837567698 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8887750545554565 = (11.670574021130504 - 1.2980589588067477) / 11.670574021130504 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 16.07 ns 0.99 ns 0.06 0.14 False
BitwiseOrBenchmark - Duration of single invocation 13.26 ns 1.00 ns 0.08 0.12 False
AllBitsSetBenchmark - Duration of single invocation 6.55 ns 1.11 ns 0.17 0.17 True
AddOperatorBenchmark - Duration of single invocation 37.95 ns 1.09 ns 0.03 0.11 False
SubtractionOperatorBenchmark - Duration of single invocation 38.15 ns 1.16 ns 0.03 0.12 False
EqualityOperatorBenchmark - Duration of single invocation 8.27 ns 0.11 ns 0.01 0.16 False
OnesComplementBenchmark - Duration of single invocation 11.14 ns 1.23 ns 0.11 0.16 False
EqualsAllBenchmark - Duration of single invocation 8.21 ns 0.15 ns 0.02 0.14 False
SubtractBenchmark - Duration of single invocation 38.00 ns 1.13 ns 0.03 0.12 False
UnaryNegateOperatorBenchmark - Duration of single invocation 35.69 ns 1.03 ns 0.03 0.11 False
OnesComplementOperatorBenchmark - Duration of single invocation 11.18 ns 1.01 ns 0.09 0.17 False
DivisionOperatorBenchmark - Duration of single invocation 41.24 ns 1.02 ns 0.02 0.09 True
MultiplyBenchmark - Duration of single invocation 38.70 ns 1.02 ns 0.03 0.06 True
InequalityOperatorBenchmark - Duration of single invocation 8.47 ns 0.10 ns 0.01 0.15 False
NegateBenchmark - Duration of single invocation 35.75 ns 1.08 ns 0.03 0.09 False
DivideBenchmark - Duration of single invocation 41.27 ns 1.06 ns 0.03 0.06 True
BitwiseAndBenchmark - Duration of single invocation 13.48 ns 1.04 ns 0.08 0.12 False
XorBenchmark - Duration of single invocation 13.11 ns 1.19 ns 0.09 0.15 False
MultiplyOperatorBenchmark - Duration of single invocation 38.85 ns 1.01 ns 0.03 0.06 True
BitwiseAndOperatorBenchmark - Duration of single invocation 13.18 ns 1.26 ns 0.10 0.12 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 13.22 ns 1.30 ns 0.10 0.14 False
AddBenchmark - Duration of single invocation 37.91 ns 1.15 ns 0.03 0.09 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9890761223798242 < 12.666572654092619.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.34342790581603 (T) = (0 -1.188465742903994) / Math.Sqrt((15.990189849323057 / (299)) + (0.06513154214999282 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9061265829558738 = (12.66030129003765 - 1.188465742903994) / 12.66030129003765 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.BitwiseOrBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0028286319210087 < 12.529268541163265.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.477014260180184 (T) = (0 -1.142870089685792) / Math.Sqrt((19.107202414009823 / (299)) + (0.016283791428929762 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9151239043575183 = (13.465158606020616 - 1.142870089685792) / 13.465158606020616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.AllBitsSetBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1062294821994132 < 6.205069129533133.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 146.251301947073 (T) = (0 -1.176220062691854) / Math.Sqrt((0.05020148917732999 / (299)) + (0.021816970703117385 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8059999279422705 = (6.062987761890321 - 1.176220062691854) / 6.062987761890321 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0853209231761747 < 36.01556834142325.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.330129454445185 (T) = (0 -1.2322635698813138) / Math.Sqrt((142.03487265619867 / (299)) + (0.06771167757581381 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9651283168122081 = (35.33708319283859 - 1.2322635698813138) / 35.33708319283859 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1587546294668953 < 36.06531326980353.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.416501630417976 (T) = (0 -1.1413045776004718) / Math.Sqrt((143.97553126813537 / (299)) + (0.012690014009478175 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.967807039349767 = (35.45199181896959 - 1.1413045776004718) / 35.45199181896959 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.1054924235175729 < 7.8912593632360535.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 75.78485063762689 (T) = (0 -0.06234628183141613) / Math.Sqrt((5.507314461823998 / (299)) + (0.008379985396430619 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9940358160109108 = (10.453447101141153 - 0.06234628183141613) / 10.453447101141153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2343458257604742 < 10.586151926233985.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.896238180965256 (T) = (0 -1.1295162672570132) / Math.Sqrt((7.284160645387523 / (299)) + (0.01650138385011523 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8679885232606234 = (8.556197500062504 - 1.1295162672570132) / 8.556197500062504 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.EqualsAllBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.14743217672559647 < 7.829383514760114.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 67.76812639037344 (T) = (0 -0.11352003135352291) / Math.Sqrt((8.486655855319505 / (299)) + (0.011089329053029938 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9902403893102324 = (11.631614719277925 - 0.11352003135352291) / 11.631614719277925 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1347368994046856 < 36.0422202013849.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.293393456128975 (T) = (0 -1.1637332823570201) / Math.Sqrt((180.58403195253757 / (299)) + (0.010493724154979527 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9699364033937548 = (38.70905060358854 - 1.1637332823570201) / 38.70905060358854 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0323082525772656 < 34.014965061319096.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.940602010312936 (T) = (0 -1.1869608475802438) / Math.Sqrt((96.95281901615247 / (299)) + (0.03708657365215341 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9592455440251322 = (29.124688802427162 - 1.1869608475802438) / 29.124688802427162 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0097774085693436 < 10.559453543843766.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.45510006354069 (T) = (0 -1.2197410952211838) / Math.Sqrt((7.289067080526882 / (299)) + (0.11968561553385393 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8568673966730617 = (8.521755818519523 - 1.2197410952211838) / 8.521755818519523 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0247489412870103 < 39.1343767069596.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 399.10783942602114 (T) = (0 -1.1484893609187876) / Math.Sqrt((2.9574433129130093 / (299)) + (0.017552938351105433 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9728773186709717 = (42.34424122697655 - 1.1484893609187876) / 42.34424122697655 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0245599780545975 < 36.87390975348868.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 176.04344548919656 (T) = (0 -1.2174815343637098) / Math.Sqrt((15.315230159158316 / (299)) + (0.14558679536296107 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9719797553072849 = (43.45006789609638 - 1.2174815343637098) / 43.45006789609638 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.09697877397158391 < 8.03293551525679.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 73.21016766833809 (T) = (0 -0.11665526643450362) / Math.Sqrt((5.4305669518699835 / (299)) + (0.04314392550424597 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9888688076797424 = (10.480033322414474 - 0.11665526643450362) / 10.480033322414474 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.079884526052723 < 33.9848262008158.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.91195945486629 (T) = (0 -1.1834065557575175) / Math.Sqrt((96.64003587935184 / (299)) + (0.04949315405171231 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9593094328105977 = (29.08306857088319 - 1.1834065557575175) / 29.08306857088319 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0617009064316538 < 39.151210719091374.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 169.46332506939376 (T) = (0 -1.311983334556331) / Math.Sqrt((16.493919290175356 / (299)) + (0.3259133359936361 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9714375714257699 = (45.93388587901944 - 1.311983334556331) / 45.93388587901944 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.BitwiseAndBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0401480478685807 < 12.515449959247544.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.0336713802878 (T) = (0 -1.1702792151797081) / Math.Sqrt((18.67608555922613 / (299)) + (0.022139405812413953 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9134618629422011 = (13.523277192783537 - 1.1702792151797081) / 13.523277192783537 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.XorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1854413566202044 < 12.533091898807921.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.18099934647665 (T) = (0 -1.219153807995259) / Math.Sqrt((18.992240357915996 / (299)) + (0.025138071680441812 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9094659372544724 = (13.466244317589577 - 1.219153807995259) / 13.466244317589577 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0067371288458768 < 36.84343248878254.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 369.64476243508506 (T) = (0 -1.2036411192144667) / Math.Sqrt((2.8697588601868422 / (299)) + (0.03348482680255792 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9699591834941917 = (40.06685766952262 - 1.2036411192144667) / 40.06685766952262 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2585726646020843 < 12.526482030187418.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.189713913633724 (T) = (0 -1.178625542708572) / Math.Sqrt((16.12256409459062 / (299)) + (0.013614856150188353 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9069274156579799 = (12.663509357142123 - 1.178625542708572) / 12.663509357142123 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.3019168485414492 < 12.528406544097669.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.31979799326314 (T) = (0 -1.1627695128945432) / Math.Sqrt((16.0606504167414 / (299)) + (0.017889926921979354 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.908268453001225 = (12.675786585285344 - 1.1627695128945432) / 12.675786585285344 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Single>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1455804381177663 < 36.00894923207618.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.44153575706583 (T) = (0 -1.1626979381671407) / Math.Sqrt((179.65299763369148 / (299)) + (0.012465532633163627 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.969978413159485 = (38.72873024146891 - 1.1626979381671407) / 38.72873024146891 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OneBenchmark - Duration of single invocation 11.05 ns 1.01 ns 0.09 0.35 False
AddOperatorBenchmark - Duration of single invocation 29.09 ns 1.07 ns 0.04 0.12 False
EqualityOperatorBenchmark - Duration of single invocation 5.05 ns 0.00 ns 0.00 0.23 False
MultiplyBenchmark - Duration of single invocation 30.13 ns 1.02 ns 0.03 0.06 True
AndNotBenchmark - Duration of single invocation 17.64 ns 1.06 ns 0.06 0.11 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.37 ns 1.04 ns 0.11 0.16 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.46 ns 1.03 ns 0.11 0.14 False
SubtractionOperatorBenchmark - Duration of single invocation 29.21 ns 1.09 ns 0.04 0.09 False
DivisionOperatorBenchmark - Duration of single invocation 32.78 ns 1.05 ns 0.03 0.05 True
MultiplyOperatorBenchmark - Duration of single invocation 29.99 ns 1.06 ns 0.04 0.06 True
AddBenchmark - Duration of single invocation 29.12 ns 0.97 ns 0.03 0.11 False
SubtractBenchmark - Duration of single invocation 29.29 ns 1.01 ns 0.03 0.11 False
ConditionalSelectBenchmark - Duration of single invocation 32.97 ns 0.96 ns 0.03 0.36 False
UnaryNegateOperatorBenchmark - Duration of single invocation 29.25 ns 1.04 ns 0.04 0.09 False
OnesComplementOperatorBenchmark - Duration of single invocation 8.97 ns 1.10 ns 0.12 0.13 False
NegateBenchmark - Duration of single invocation 29.38 ns 1.12 ns 0.04 0.11 False
DivideBenchmark - Duration of single invocation 32.50 ns 1.02 ns 0.03 0.06 True
BitwiseAndOperatorBenchmark - Duration of single invocation 9.58 ns 0.99 ns 0.10 0.12 False
InequalityOperatorBenchmark - Duration of single invocation 4.86 ns 0.22 ns 0.05 0.28 False
OnesComplementBenchmark - Duration of single invocation 8.79 ns 1.12 ns 0.13 0.15 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Single>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.010252239737983 < 10.459703124878159.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 31.585218688095846 (T) = (0 -1.2403903658588542) / Math.Sqrt((20.84300154445499 / (299)) + (0.029157303919878707 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8715307537072817 = (9.655154067243565 - 1.2403903658588542) / 9.655154067243565 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0731603957964606 < 27.644206856758277.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.0946100217073 (T) = (0 -1.1039575812023106) / Math.Sqrt((60.396960732627576 / (299)) + (0.0064895287063534285 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9523837180778285 = (23.184455749962257 - 1.1039575812023106) / 23.184455749962257 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.EqualityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0 < 4.686477406138721.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 62.5265231257179 (T) = (0 -0.07534204430172356) / Math.Sqrt((3.5670842379969927 / (299)) + (0.02959892349082666 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9896276019239324 = (7.263705437179538 - 0.07534204430172356) / 7.263705437179538 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0247269198176574 < 28.56319031006255.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 289.96949320343265 (T) = (0 -1.1852853953439553) / Math.Sqrt((0.8803812410742657 / (299)) + (0.13153336221759052 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9579308171116038 = (28.174671195500835 - 1.1852853953439553) / 28.174671195500835 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0577056201004447 < 16.70983967550056.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.44149789702103 (T) = (0 -1.104955179789576) / Math.Sqrt((51.87341965492458 / (299)) + (0.005928529577154546 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9448799713552023 = (20.046346254826616 - 1.104955179789576) / 20.046346254826616 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0448080037845986 < 8.91355803273863.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.74756923559313 (T) = (0 -1.1377706802702716) / Math.Sqrt((16.1902721650217 / (299)) + (0.06146283038472635 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8995569338180496 = (11.32751839942068 - 1.1377706802702716) / 11.32751839942068 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0254015758292023 < 9.123020374121309.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.29667912053534 (T) = (0 -1.2397315948947023) / Math.Sqrt((16.130944469436926 / (299)) + (0.13201504657688273 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8905710142004126 = (11.329097001458058 - 1.2397315948947023) / 11.329097001458058 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0948025987568406 < 27.632034561310128.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.59668812071798 (T) = (0 -1.1989800294703616) / Math.Sqrt((60.16212692390185 / (299)) + (0.06859004622907679 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9482277329549654 = (23.158731458049136 - 1.1989800294703616) / 23.158731458049136 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.DivisionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0504691487353677 < 30.841898872018476.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 269.91769462867575 (T) = (0 -1.1751108209009808) / Math.Sqrt((3.0785322156326216 / (299)) + (0.012270445569337863 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9598447427528585 = (29.264183607854466 - 1.1751108209009808) / 29.264183607854466 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0557494015100939 < 28.586507364659123.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 294.3269977914741 (T) = (0 -1.1376458438594756) / Math.Sqrt((2.062128595711911 / (299)) + (0.024773990844567908 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9585104288171846 = (27.420043433244278 - 1.1376458438594756) / 27.420043433244278 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9670815582553184 < 27.630993334695948.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.60158056935345 (T) = (0 -1.1383647014660085) / Math.Sqrt((65.0165599748777 / (299)) + (0.012626307036334885 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9531485975385344 = (24.297345258816858 - 1.1383647014660085) / 24.297345258816858 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0076610369956103 < 27.68135035152784.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 49.62442074946206 (T) = (0 -1.146965799186609) / Math.Sqrt((64.97127380039348 / (299)) + (0.009862310138637302 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9528039773732112 = (24.30217072011453 - 1.146965799186609) / 24.30217072011453 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.959985900938347 < 31.30385003474121.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.27102907912047 (T) = (0 -1.1498849480583442) / Math.Sqrt((365.8759143665352 / (299)) + (0.1370567684183585 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9754898683756825 = (46.91467861875935 - 1.1498849480583442) / 46.91467861875935 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.043177493199453 < 27.875397810394258.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.552556838497985 (T) = (0 -1.1173817966266681) / Math.Sqrt((61.944304622923646 / (299)) + (0.011992174950944573 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9519290029382247 = (23.244406501299224 - 1.1173817966266681) / 23.244406501299224 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0959487795270622 < 8.627066160154413.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.94966678622078 (T) = (0 -1.1591230582495935) / Math.Sqrt((5.5809698821365155 / (299)) + (0.011734280818350045 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8459034028777532 = (7.522054866208674 - 1.1591230582495935) / 7.522054866208674 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1195607386802995 < 27.818435235397754.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.53818788681897 (T) = (0 -1.2276795501774518) / Math.Sqrt((65.23351746244576 / (299)) + (0.08084723021420694 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9490192763371171 = (24.081249970001508 - 1.2276795501774518) / 24.081249970001508 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.DivideBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0191263339190342 < 30.782829357533654.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 401.2407511170766 (T) = (0 -1.170140155891863) / Math.Sqrt((1.3402632952411833 / (299)) + (0.016819505544040568 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9612978648475131 = (30.234511643388615 - 1.170140155891863) / 30.234511643388615 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9850850197155193 < 9.087644459019222.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.73391162535451 (T) = (0 -1.1206149598909931) / Math.Sqrt((16.066730597488206 / (299)) + (0.019359254171870434 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9011591678492722 = (11.33757107773137 - 1.1206149598909931) / 11.33757107773137 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.22244670062417002 < 4.624246116396517.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 65.63143082013616 (T) = (0 -0.04975733417374043) / Math.Sqrt((3.505196033801136 / (299)) + (0.005741841756926974 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9931190196336628 = (7.231140262681375 - 0.04975733417374043) / 7.231140262681375 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Single>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1196270451561654 < 8.616214800010406.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.9174153216137 (T) = (0 -1.1507238645294533) / Math.Sqrt((5.633219120238884 / (299)) + (0.009475571659482237 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8470161964354856 = (7.521867267760704 - 1.1507238645294533) / 7.521867267760704 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 9.50 ns 1.12 ns 0.12 0.14 False
OneBenchmark - Duration of single invocation 5.74 ns 1.21 ns 0.21 0.30 False
ConditionalSelectBenchmark - Duration of single invocation 32.82 ns 1.01 ns 0.03 0.35 False
OnesComplementOperatorBenchmark - Duration of single invocation 8.77 ns 1.06 ns 0.12 0.18 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.54 ns 1.00 ns 0.10 0.12 False
OnesComplementBenchmark - Duration of single invocation 9.17 ns 0.98 ns 0.11 0.17 False
AndNotBenchmark - Duration of single invocation 17.66 ns 1.02 ns 0.06 0.09 False
BitwiseOrOperatorBenchmark - Duration of single invocation 9.59 ns 1.08 ns 0.11 0.13 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1152195887562462 < 9.102688779752945.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.88316237682448 (T) = (0 -1.2256458386576279) / Math.Sqrt((16.172842232887465 / (299)) + (0.09633737489300231 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8918861556786153 = (11.336622486702185 - 1.2256458386576279) / 11.336622486702185 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.207363262377634 < 5.54078713967165.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 29.588957218517493 (T) = (0 -1.289226916384479) / Math.Sqrt((9.755755988175489 / (299)) + (0.025042101315622785 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8082188448166236 = (6.722385810804779 - 1.289226916384479) / 6.722385810804779 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.005026018167203 < 31.146610066789968.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.53738260231109 (T) = (0 -1.0992953805005816) / Math.Sqrt((358.81502336889184 / (299)) + (0.024415977788517854 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.976421249878185 = (46.622292310715494 - 1.0992953805005816) / 46.622292310715494 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.062626449903031 < 8.608483426243263.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.582457762215846 (T) = (0 -1.1726857082646225) / Math.Sqrt((5.650069939291198 / (299)) + (0.010605282478631099 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8439468235521265 = (7.514654523269733 - 1.1726857082646225) / 7.514654523269733 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.99682157486541 < 9.013208898625326.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.85647619496421 (T) = (0 -1.1134456725954107) / Math.Sqrt((16.06361956341404 / (299)) + (0.010174177940904493 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9016434274888516 = (11.320500950449501 - 1.1134456725954107) / 11.320500950449501 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9817457618989923 < 8.644797905872787.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.4164385822807 (T) = (0 -1.1861585725799106) / Math.Sqrt((5.541920159186196 / (299)) + (0.019364645671558704 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8420172294491832 = (7.508151480343677 - 1.1861585725799106) / 7.508151480343677 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0209594389874934 < 16.762803839115566.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.377191061630604 (T) = (0 -1.1269290556947584) / Math.Sqrt((51.865593419242344 / (299)) + (0.007975391089328801 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9437798573459937 = (20.04493411961225 - 1.1269290556947584) / 20.04493411961225 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<Int64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0793403037583973 < 9.088040052525947.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.72548658916773 (T) = (0 -1.1242027458092374) / Math.Sqrt((16.13944838962675 / (299)) + (0.008819065345962109 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.900680416163016 = (11.319044063398636 - 1.1242027458092374) / 11.319044063398636 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 11, 2023 •
edited

Run Information

Name Value
Architecture arm64
OS ubuntu 20.04
Queue AmpereUbuntu
Baseline 8b1d1eabe32ba781ffcce2867333dfdc53bdd635
Compare e0c94f84b47fdd48435a625446123f89268b9c20
Diff Diff
Configs CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Improvements in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 9.58 ns 1.01 ns 0.11 0.15 False
UnaryNegateOperatorBenchmark - Duration of single invocation 10.67 ns 1.07 ns 0.10 0.12 False
ConditionalSelectBenchmark - Duration of single invocation 32.92 ns 1.11 ns 0.03 0.35 False
OnesComplementOperatorBenchmark - Duration of single invocation 9.18 ns 1.28 ns 0.14 0.14 False
SubtractionOperatorBenchmark - Duration of single invocation 10.69 ns 1.07 ns 0.10 0.14 False
MultiplyOperatorBenchmark - Duration of single invocation 11.07 ns 1.16 ns 0.11 0.09 True
BitwiseOrOperatorBenchmark - Duration of single invocation 9.53 ns 1.07 ns 0.11 0.15 False
OnesComplementBenchmark - Duration of single invocation 9.01 ns 1.08 ns 0.12 0.15 False
AndNotBenchmark - Duration of single invocation 17.68 ns 1.04 ns 0.06 0.13 False
OneBenchmark - Duration of single invocation 9.00 ns 1.22 ns 0.14 0.33 False
AddBenchmark - Duration of single invocation 10.86 ns 0.99 ns 0.09 0.12 False
NegateBenchmark - Duration of single invocation 11.09 ns 1.15 ns 0.10 0.11 False
MultiplyBenchmark - Duration of single invocation 11.03 ns 1.04 ns 0.09 0.08 True
SubtractBenchmark - Duration of single invocation 10.80 ns 1.12 ns 0.10 0.13 False
AddOperatorBenchmark - Duration of single invocation 11.08 ns 0.99 ns 0.09 0.12 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 9.52 ns 1.03 ns 0.11 0.13 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0060859092702796 < 9.102710834638344.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.69172304016606 (T) = (0 -1.1215842748190703) / Math.Sqrt((16.08856739188478 / (299)) + (0.014363902429821452 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9008780447118715 = (11.315195221470768 - 1.1215842748190703) / 11.315195221470768 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.UnaryNegateOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.068871947799924 < 10.197940155901296.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.200824847505366 (T) = (0 -1.1430746381801786) / Math.Sqrt((10.757615734196433 / (299)) + (0.01862902285395077 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8899637517969168 = (10.388164417152044 - 1.1430746381801786) / 10.388164417152044 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.ConditionalSelectBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1136199482225264 < 31.190737492989246.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 41.303817105829225 (T) = (0 -1.1686019266010865) / Math.Sqrt((364.38731413948545 / (299)) + (0.1272537877022563 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9750666252644956 = (46.86898340067186 - 1.1686019266010865) / 46.86898340067186 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.OnesComplementOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2780429881177606 < 8.646500611523468.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 46.170102572270395 (T) = (0 -1.1357605362982457) / Math.Sqrt((5.594380259645179 / (299)) + (0.007567574785888761 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8486954423905961 = (7.506452906925888 - 1.1357605362982457) / 7.506452906925888 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.068808248405259 < 10.180242846072483.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 44.950227861688504 (T) = (0 -1.2939919436149) / Math.Sqrt((10.879456612227251 / (299)) + (0.16915680264744684 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8790098232018717 = (10.695016553070435 - 1.2939919436149) / 10.695016553070435 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.MultiplyOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1643150078783584 < 10.527685627205.
IsChangePoint: Marked as a change because one of 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 112.56288978539975 (T) = (0 -1.2160756627761233) / Math.Sqrt((0.18658207233256327 / (299)) + (0.1983908316448334 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.899012615889441 = (12.041857242730323 - 1.2160756627761233) / 12.041857242730323 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.073551418180387 < 9.02410440779617.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 43.96483824667346 (T) = (0 -1.1046898522839468) / Math.Sqrt((16.05526737274017 / (299)) + (0.0053176524195028925 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.902363894136634 = (11.314357967429313 - 1.1046898522839468) / 11.314357967429313 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.OnesComplementBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0829400164653953 < 8.60081149630128.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.34539663555495 (T) = (0 -1.1571374806815056) / Math.Sqrt((5.617304521704531 / (299)) + (0.01992968386766101 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8460054982559883 = (7.514148022018602 - 1.1571374806815056) / 7.514148022018602 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.AndNotBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.042697804176136 < 16.755661753216543.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 45.51455953651413 (T) = (0 -1.1102457996372241) / Math.Sqrt((51.66561684684393 / (299)) + (0.008197053690973274 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.9446272222130524 = (20.05039017383246 - 1.1102457996372241) / 20.05039017383246 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.OneBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.2182211230405287 < 8.26473694690517.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 30.934875753858098 (T) = (0 -1.2226125122659286) / Math.Sqrt((16.6844366961198 / (299)) + (0.0102031002087453 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8571563765768888 = (8.559097584947695 - 1.2226125122659286) / 8.559097584947695 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.AddBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9860357713181541 < 10.266838124526286.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.803593685127595 (T) = (0 -1.1254106971047626) / Math.Sqrt((13.893175562530711 / (299)) + (0.008976339582556394 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9037248841907404 = (11.689528365091011 - 1.1254106971047626) / 11.689528365091011 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.NegateBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.146352770536958 < 10.33673275202075.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.02464071524457 (T) = (0 -1.1746472373228978) / Math.Sqrt((12.245780468478074 / (299)) + (0.00893801985567747 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8926248069279771 = (10.939651922534766 - 1.1746472373228978) / 10.939651922534766 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.MultiplyBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.0406595315816725 < 10.497477292157845.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 107.17271416097635 (T) = (0 -1.2348460954630263) / Math.Sqrt((1.0304238968549029 / (299)) + (0.20094586655307609 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9054787304616931 = (13.064214028172547 - 1.2348460954630263) / 13.064214028172547 is greater than 0.05.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.SubtractBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.1186994270278185 < 10.22948869295336.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.09689830878424 (T) = (0 -1.1283785472301433) / Math.Sqrt((14.11291167259068 / (299)) + (0.01850121624630808 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.9032796698128869 = (11.666405036533753 - 1.1283785472301433) / 11.666405036533753 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.AddOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 0.9852174806015588 < 10.305598995308962.
IsChangePoint: Marked as a change because one of 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 48.90876860727581 (T) = (0 -1.1987765010289504) / Math.Sqrt((11.000399065375424 / (299)) + (0.021884523453932363 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8879694897367575 = (10.700446674857924 - 1.1987765010289504) / 10.700446674857924 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Numerics.Tests.Perf_VectorOf<UInt32>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed: Marked as improvement because 1.034942601427347 < 9.066251950557513.
IsChangePoint: Marked as a change because one of 3/6/2023 4:10:52 PM, 3/15/2023 10:57:55 AM, 3/24/2023 12:04:20 PM, 4/3/2023 11:26:06 PM, 4/11/2023 5:52:15 AM falls between 4/2/2023 3:55:31 PM and 4/11/2023 5:52:15 AM.
IsImprovementStdDev: Marked as improvement because 42.42390362088458 (T) = (0 -1.174529658490979) / Math.Sqrt((16.23356259094096 / (299)) + (0.06149746621860522 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.8960605615128656 = (11.300134728324142 - 1.174529658490979) / 11.300134728324142 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@lewing
Copy link
Member

lewing commented Apr 11, 2023

looks like this is dotnet/runtime#84180

cc @fanyang-mono @SamMonoRT

@kotlarmilos
Copy link
Member

Similar to #15718.

@kotlarmilos kotlarmilos mentioned this issue Apr 12, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-improvement runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lewing @kotlarmilos