ci: run Android instrumented tests on the mac-metal queue#471
Merged
ci: run Android instrumented tests on the mac-metal queue#471
Conversation
Runs :app:connectedDebugAndroidTest against an ephemeral AVD booted via the start-android-emulator helper on the mac-metal agent. See: https://appsinfrap2.wordpress.com/2025/10/10/using-the-mac-metal-queue-for-android-instrumented-tests/
github-actions
Bot
added
the
[Type] Build Tooling
These were lost during a merge conflict resolution in #339 and the gap was invisible until the mac-metal CI step started exercising the androidTest sources. Re-adds espresso-web, compose-ui-test-junit4, and compose-ui-test-manifest to the version catalog and wires them into the app module so EditorInteractionTest compiles.
Registers the stock InstrumentedTestCollector as a test listener when BUILDKITE_ANALYTICS_TOKEN_ANDROID is present in the environment. The listener is only attached when the token is set, so local runs and unauthenticated CI stay fully inert. The Android Test Engine suite + agent secret need to be set up separately for data to start flowing.
…_ANDROID_E2E Matches the env var name already provisioned on Automattic agents rather than inventing a new one.
iangmaia
reviewed
Apr 24, 2026
| make test-ios-e2e | ||
| plugins: *plugins | ||
|
|
||
| - label: ':android: Test Android E2E' |
There was a problem hiding this comment.
Not a blocker as this is just the initial setup to get it going, but maybe for the long run it might be worth it to add artifact_paths to debug test results (stuff from logcat, test reports, etc).
iangmaia
reviewed
Apr 24, 2026
| System.getenv("BUILDKITE_ANALYTICS_TOKEN_ANDROID_E2E")?.takeIf { it.isNotBlank() }?.let { token -> | ||
| testInstrumentationRunnerArguments["listener"] = | ||
| "com.buildkite.test.collector.android.InstrumentedTestCollector" | ||
| testInstrumentationRunnerArguments["BUILDKITE_ANALYTICS_TOKEN"] = token |
There was a problem hiding this comment.
By the way, it's probably also interesting to set these for Buildkite's test collector https://github.com/buildkite/test-collector-android/blob/main/CI_CONFIGURATION.md#buildkite
Pass BUILDKITE_BUILD_ID/URL/BRANCH/COMMIT/BUILD_NUMBER/JOB_ID/MESSAGE through to the collector so uploaded test results link back to the Buildkite job. Per buildkite/test-collector-android CI_CONFIGURATION.md.
Pick up Gradle's JUnit XML, HTML reports, and any additional emulator outputs (logcat captures, screenshots) from connectedDebugAndroidTest. Artifacts upload on both success and failure, so the reports are available directly from the Buildkite UI when a test run goes wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds a new Buildkite step that runs
:app:connectedDebugAndroidTeston themac-metalqueue, booting an ephemeral AVD via thestart-android-emulatorhelper.Why?
The Android instrumented tests in
android/app/src/androidTest/haven't had a CI job — they've only run locally. The newmac-metalagents can boot real Android emulators (something the virtualizedmacqueue can't do), so we can finally wire these tests into the pipeline.How?
dist.tar.gzartifact produced byBuild React Appso the editor assets are available.start-android-emulatorwithsystem-images;android-34;google_apis;arm64-v8a(pre-installed on the agent, matches ourtargetSdk = 34) and apixel_5device profile.make test-android-e2e.ENSURE_ANDROID_DEVICEdetects the already-booted emulator viaadband skips its fallback boot path, so the existing Makefile target works unchanged.pre-exithook — no extra teardown needed here.Test plan
:android: Test Android E2Estep on this PR against themac-metalqueue and the step succeeds.EditorInteractionTesttests (testEditorWebViewBecomesVisible,testUndoRedoAfterTyping) pass on CI.