[Android][infra] Increase the Android test coverage across different OS versions

[matouskozak]

For .NET 9, we support Android API version of 21 and newer (https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.md#android). Our CI infrastructure only test a fixed API version:

runtime/eng/pipelines/libraries/helix-queues-setup.yml

Lines 94 to 98 in 9a18cf5

	# Android
	- ${{ if in(parameters.platform, 'android_x86', 'android_x64', 'linux_bionic_x64') }}:
	- Ubuntu.2204.Amd64.Android.29.Open
	- ${{ if in(parameters.platform, 'android_arm', 'android_arm64', 'linux_bionic_arm', 'linux_bionic_arm64') }}:
	- Windows.11.Amd64.Android.Open

We should explore the possibility of having coverage over both older and newer Android API version for the Libraries tests.

TODO:

• Identify which Libraries tests should be evaluated on different API versions (prone to breaking Android API changes)
• Estimate cost of new CI infrastructure jobs for Android testing
• Introduce new Android jobs covering older/newer OS versions

[dotnet-policy-service]

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

[matouskozak]

fyi: @vitek-karas @simonrozsival

[akoeplinger]

We do have Helix queues for the newer API levels, it just never was worth the CI cost and pain of triaging flaky issues to keep them enabled all the time. We used a temporary PR (#64341) to trigger the "full" set from time to time, but nowadays having a separate pipeline probably makes more sense.

[matouskozak]

We do have Helix queues for the newer API levels, it just never was worth the CI cost and pain of triaging flaky issues to keep them enabled all the time. We used a temporary PR (#64341) to trigger the "full" set from time to time, but nowadays having a separate pipeline probably makes more sense.

I think a temporary PR might be a good idea as well. Our initial idea was to do ~weekly runs but as you mentioned it depends on the CI cost and flakiness of the issues. I think it also depends on how many really issues we can discover using this approach and perhaps just a selected set of Libraries test could be run if they are prone to breaking changes by Android OS bumps.

[akoeplinger]

perhaps just a selected set of Libraries test could be run if they are prone to breaking changes by Android OS bumps.

IMO trying to guess which tests could be broken is kinda impossible, much easier to run everything. If you only run weekly it should be manageable to deal with the flaky issues.