[Perf -118%] System.Collections.Tests.Perf_BitArray for small inputs (3)

[DrewScoggins]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
BitArrayNot	1.25 ns	2.34 ns	1.87		False
BitArrayIntArrayCtor	143.62 ns	151.32 ns	1.05		False
BitArraySetAll	1.56 ns	5.66 ns	3.62		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Collections.Tests.Perf_BitArray*';

Histogram

System.Collections.Tests.Perf_BitArray.BitArrayNot(Size: 4)

[0.372 ; 1.006) | @@@@@@@@@@@@@@
[1.006 ; 1.753) | @@@@@@@
[1.753 ; 2.387) | @@@@@@@@@@@
[2.387 ; 3.012) | @@@

System.Collections.Tests.Perf_BitArray.BitArrayIntArrayCtor(Size: 512)

[136.771 ; 143.213) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[143.213 ; 149.753) | @@
[149.753 ; 153.484) | @@@@@@@@
[153.484 ; 157.663) | @@@@@@@@@@@

System.Collections.Tests.Perf_BitArray.BitArraySetAll(Size: 4)

[0.647 ; 2.210) | @@@@@@@@@@@@@@@@
[2.210 ; 3.772) | 
[3.772 ; 4.562) | 
[4.562 ; 6.125) | @@@@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[ghost]

Tagging subscribers to this area: @eiriktsarpalis
Notify danmosemsft if you want to be subscribed.

[DrewScoggins]

@adamsitnik

[danmoseley]

@DrewScoggins do I understand correctly that the regression here is between the last datapoint before the repo consolidation and the first datapoint after ? It seems the regression was at that time, and there was some overlap (consistent with it being the old and new infrastructure)?

I was trying to get something useful out of the diff link, but it just encompasses 6 months worth of work so it isn't useful. But even scoping to the point of the regression, if the regression is across the consolidation point, the product code may not even have changed (?).

The nearest potentially relevant BitArray change entered corefx master in Nov.

[DrewScoggins]

As you can see from the graph it does look like this regression came in in November, so good tracking down to that commit.

Historical Data in Reporting System

[danmoseley]

Ah good that helps. In the original PR, @Gnbrkm41 noted that there was some regression for the smallest size (which is non vectorized). These scenarios were added at the same time it looks like (dotnet/performance#1026)

I think we figure that's OK - these are tiny bit arrays and tiny absolute times. @DrewScoggins is there still a regression in the int ctor 512 size? You reported it above, but it's not in the latest report

[DrewScoggins]

That is because it is floating right around the regression threshold. So in the latest run it just slightly dropped, but that is enough to move it below 5%. Given that we have many other issues with much bigger regressions, we can probably leave this for now and if we see it come back we can take another look then.

[danmoseley]

OK I'm going to close this then as acceptable/known based on current data and discussion in original PR.

cc @eiriktsarpalis who owns collections