Setup emulator WTF

[TimoPtr]

Summary

This PR integrates emulator.wtf suing the Github Action to run instrumentation tests across all supported Android versions in CI. The token is injected using the Github environment ui-test.

Test execution is significantly faster: our 4 instrumentation tests now run across 14 emulators in under 30 seconds (excluding the build that is still on GHA).

As part of this change, Android phones emulators have been removed from GitHub Actions since they are now handled by emulator.wtf. Wear OS and Automotive emulators are still executed in CI to ensure the app launches correctly on those platforms.

Thanks to the emulator.wtf team for providing such a useful tool.

We will monitor this setup closely. If it introduces issues either on our side or theirs we can revert this PR and fall back to the previous configuration.

In case of failures, results will continue to be reported in PR comments as before, with the addition of a link to emulator.wtf containing the full logcat output.

I've added a Gradle flag abis to select the ABI we want to build to limit the build to specific ABIs.

Checklist

• New or updated tests have been added to cover the changes following the testing guidelines.
• The code follows the project's code style and best_practices.
• The changes have been thoroughly tested, and edge cases have been considered.
• Changes are backward compatible whenever feasible. Any breaking changes are documented in the changelog for users and/or in the code for developers depending on the relevance.

Screenshots

Link to pull request in documentation repositories

Developer Documentation: home-assistant/developers.home-assistant#3072

Any other notes

The next step would be to make a full 2e2 test by starting HA in GHA and then using the egress-tunnel of the Action to access HA from the emulator.

[github-actions]

Test Results

  233 files    237 suites   9m 20s ⏱️
1 838 tests 1 838 ✅ 0 💤 0 ❌
1 873 runs  1 873 ✅ 0 💤 0 ❌

Results for commit 75174af.

♻️ This comment has been updated with latest results.

[jpelgrom]

Why did you decide to use a Gradle plugin + custom comment logic here? They offer a GitHub Action + standard junit test results we can upload using other well known actions, even mentioned in their guide.

I'm not a big fan of using a Gradle plugin which then downloads their CLI, if we can also use a lightweight action which downloads their CLI but doesn't impact local dev environments.

[TimoPtr]

Why did you decide to use a Gradle plugin + custom comment logic here? They offer a GitHub Action + standard junit test results we can upload using other well known actions, even mentioned in their guide.

I'm not a big fan of using a Gradle plugin which then downloads their CLI, if we can also use a lightweight action which downloads their CLI but doesn't impact local dev environments.

I used the plugin since it was easier for me to test on my setup. But indeed we can use the Action.

[TimoPtr]

@jpelgrom I've migrated to the action so we don't have a new deps within Gradle locally. The only downside for now is that I'm not able to send a message with the URL within the PR but I think it is acceptable since a simple click on the task would show the link within the logs.

[jpelgrom]

A lot cleaner and more consistent now <3

Minor comments to check for you but approach look good. To merge you'll need to change the required CI checks.

(Would be nice if their action also pinned dependencies using the hash instead of tag. Do you have a direct line to pass the feedback or should we submit a PR?)

[jpelgrom]

This path is no longer relevant I think, you're saving in build/test-results?

[TimoPtr]

Indeed it need to be **/build/test-results/**/results.xml I'll keep it alongside the one above just to avoid a more complex regex or catching unwanted files.

[TimoPtr]

I've opened a PR emulator-wtf/run-tests#19 on their repo for pinning the versions.

[jpelgrom]

It looks like we forgot to consider PRs from forks... 🤦

Those don't have access to secrets so the action will fail by default. Do you know if it'll work if we check (there's no malicious intent/exfiltration of the secret) and retry?