`AllocationQuantumIsNotAnIssueForNetCore21Plus` tests randomly failed on `macos (arm64)`

[filzrev]

AllocationQuantumIsNotAnIssueForNetCore21Plus test is recently enabled by PR #2562

But it's randomly failed on CI.
Because extra memory allocation is reported.
Currently it's confirmed only on macos(arm64) environment . (But it might be occurred on another arm64 images on windows/linux)

Error Log

2025-06-15T05:07:55.9366620Z [xUnit.net 00:09:48.63]     AllocationQuantumIsNotAnIssueForNetCore21Plus(toolchain: .NET 8.0) [FAIL]
2025-06-15T05:07:55.9368370Z [xUnit.net 00:09:48.63]       Assert.Equal() Failure: Values differ
2025-06-15T05:07:55.9368720Z [xUnit.net 00:09:48.63]       Expected: 88
2025-06-15T05:07:55.9368990Z [xUnit.net 00:09:48.63]       Actual:   760
...
2025-06-15T05:07:55.9500980Z [xUnit.net 00:09:48.63]         | Method              | Mean     | Error | Allocated |
2025-06-15T05:07:55.9501490Z [xUnit.net 00:09:48.63]         |-------------------- |---------:|------:|----------:|
2025-06-15T05:07:55.9502130Z [xUnit.net 00:09:48.63]         | SixtyFourBytesArray | 551.2 ms |    NA |     760 B |

If it can't determine root cause of this memory allocation issue.
This test should be temporary skipped for ARM64 environment.

[timcassell]

Hm... My guess is because it's a long-running benchmark, it's not able to get up to tier1 JIT before the measurement happens. It's not an uncommon thing to happen in real-world benchmarks. Maybe could try adding [MethodImpl(CodeGenHelper.AggressiveOptimization)] to it? (I know most real benchmarks won't do that, but I'm not sure what else we can do.)

[timcassell]

Actually I realized we specifically disable tiered JIT for these tests for exactly that reason. Then I'm really not sure what the cause is...

[filzrev]

I've confirmed it can be reproduced on macos-13 (x64) runner image also (It's occurred 3 times on 100 runs )
And it's not occurred when using windows(x64/arm64) and ubuntu(x64/arm64) images. (It's also testes 100 times)

So it seems to be a macos specific issues.

I've tested AggressiveOptimization options but it seems no resolve issue.
And it need 2 warmup executions are required to gets stable benchmark results on macos.

[filzrev]

Flakey test is fixed by adding warmupCount=2 settings.
But currently root cause of problem is not determined.

I leave my notes below for future reference.

---

1 Unexpected memory allocation reported issue is occurred on macos (x64/arm64) environment only.
On other OS environment, it successfully pass 100 test executions.
2. It's occurred only when target invocation count is too small. and test execution time is relatively large (iteration execution auto adjustment for [<100ms] is not used)
3. WarmupCount=1 setting is not enough to get stable result. (1/100 tests faild)
4. AggressiveOptimization setting seems has no effects.
5. Jit related counter (e,g, JitInfo.GetCompiledMethodCount) value is same on both success/failed tests.

[timcassell]

It would be nice to get a profile snapshot to see where those extra allocations are coming from.

[filzrev]

This problem reappeared on latest CI build.
https://github.com/dotnet/BenchmarkDotNet/actions/runs/15825847691/job/44605715048

It might be better to increase warmupCount to 3 if problem still reproduced.