[Mono][Android] Some runtime test groups are too big to run on Android devices

[fanyang-mono]

Some runtime test groups are too big to run on Android devices. Need to split them up. For example,

• JIT.HardwareIntrinsics
• JIT.Methodical.eh
• JIT.IL_Conformance
• JIT
• JIT.Regression
• PayloadGroup0

[ghost]

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Some runtime test groups are too big to run on Android devices. Need to split them up. For example,

• JIT.HardwareIntrinsics
• JIT.Methodical.eh
• JIT.IL_Conformance
• JIT

Author:	fanyang-mono
Assignees:	fanyang-mono
Labels:	area-Infrastructure
Milestone:	6.0.0

[naricc]

What do we actually have to do to split them up? Is it just a logical thing inside the msbuild files somewhere, or are a reorganization of the directory structure in /src/tests/JIT?

[trylek]

@naricc It's just a "logical thing inside the msbuild files", mostly driven by the file

https://github.com/dotnet/runtime/blob/master/src/tests/Common/testgrouping.proj

According to my recollection @echesakovMSFT implemented or last updated this logic last year more or less on experimental basis to tune (then) CoreCLR repo test performance.

This is known to be maintenance pain and fragile scheme w.r.t. new platform bring-up (as Fan's efforts just confirmed) and we're generally hoping to get rid of that at some point, either by automating the splitting or by making it unnecessary, perhaps thanks to some future changes in the Helix protocol that would reduce the work item overhead within a single run, @MattGal might know whether something like this could be doable.

The CoreCLR test portfolio is also generally believed to be poorly understood in terms of code coverage of the runtime and other parts of the system; I believe that as a medium-term goal we should design a plan to audit the CoreCLR test set and measure some form of "cost / effect" ratio with the aim to optimize the set for performance without degrading the general level of testing.

[fanyang-mono]

I agree with your idea that we should audit the runtime tests, in general. When should we aim on doing that? Also, I would like to know what we need to get ready before getting rid of this manually split scheme?