Regressions in System.Collections.Tests.Perf_BitArray

[performanceautofiler]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Baseline	a5fda753afce966b9d31c76559636eb515bfa05a
Compare	1df36c702a46363336b6ea5d0d9558d513e7be8e
Diff	Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
BitArrayBoolArrayCtor - Duration of single invocation	31.59 ns	40.88 ns	1.29	0.13	True

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayBoolArrayCtor(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.87508460982235 > 33.235490963584866.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 2/8/2022 12:37:33 AM falls between 1/30/2022 3:20:30 AM and 2/8/2022 12:37:33 AM.
IsRegressionStdDev: Marked as regression because -22.70736782183897 (T) = (0 -40.01058690490022) / Math.Sqrt((5.070068259802953 / (46)) + (0.15899135968215822 / (32))) is less than -1.9916726096441384 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (32) - 2, .025) and -0.2385693211100522 = (32.30387368955759 - 40.01058690490022) / 32.30387368955759 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[kunalspathak]

Possibly introduced in #63722

[tannergooding]

@kunalspathak, this looks like a fairly noisy test and like the current numbers are inline with the normal bimodality.

Am I missing something here?

[kunalspathak]

This is the local report...If you cannot reproduce it, please close it.

[tannergooding]

Not able to reproduce this locally and the current numbers look inline with the normal bi-modality

We should re-open if numbers consistently stay up at this level.

[danmoseley]

It doesn't look like the regular bimodality to me, assuming I'm reading the graph right?

[tannergooding]

Bi-modality is weird and it can show in multiple ways.

It's entirely possible that this perf machine is now stabilizing on the upper bound of the bi-modal range due to the change and that my machine simply doesn't show it and only shows an improvement.

It's also possible that this is a case where its staying on the upper bound for more runs, which its done in the past as well.

[tannergooding]

I don't think there is really anything actionable here without having runs from another week or so to see if it has actually stabilized and at that point seeing if its possibly due to alignment or something else, like a single instruction re-ordering causing the result to stabilize on these machines.

[kunalspathak]

Did you verify if there is any asmdiff before vs. after your change ?

[tannergooding]

Yes and several of the diffs are listed in the PR. There is nothing out of the ordinary and this is at worst likely a case of a microarchitectural specific differences. It may also still just be noise or having stabilized the bi-modality of the original instruction sequence.

Its a downside of us largely only testing on a single pool of machines and trying to overly normalize results. It means that we aren't accounting for cases where something may be better/worse between Haswell vs Skylake or Skylake vs Zen, etc.