[WIP] Re-enable Android tests in CIBW workflow

[rwgk]

Problem description

We recently had to disable Android tests in the cibuildwheel CI workflow because of persistent Android emulator / managed-device failures on GitHub Actions runners:

• chore(deps): bump the actions group with 5 updates #5912

This issue is to document the situation and track restoring full Android test coverage.

See also: PR #5911, where we tried to use an older Ubuntu image (commit de84648), to no avail.

Summary

• The CIBW workflow includes an Android wheel job that builds and tests pybind11_tests wheels for Android using pypa/cibuildwheel.
• Starting around mid-November 2025, the Android wheel ubuntu-latest job began failing consistently with a Gradle managed-device / emulator snapshot error.
• The failures appear to be caused by changes in the GitHub Actions ubuntu-24.04 runner image and/or Android tooling, not by changes in pybind11 itself.
• To keep CI usable while we investigate, we have temporarily disabled the Android tests in that job while still building the wheel.

Affected job and configuration

• Workflow: .github/workflows/tests-cibw.yml

• Job: currently Android wheel ubuntu-latest

• cibuildwheel: currently pinned to pypa/cibuildwheel@v3.3

• Android tests are currently disabled by setting:

  - uses: pypa/cibuildwheel@v3.3
    env:
      CIBW_PLATFORM: android
      CIBW_TEST_COMMAND: ""  # Temporarily disable Android tests; emulator setup is broken.
    with:
      package-dir: tests

Previously, cibuildwheel was running our pytest tests suite inside an Android Gradle testbed app on an emulator (via managed devices).

Failure details

The relevant error from the failing runs looks like this (truncated for brevity):

Execution failed for task ':app:maxVersionSetup'.
A failure occurred while executing ManagedDeviceSetupRunnable
com.android.build.gradle.internal.AvdSnapshotHandler$EmulatorSnapshotCannotCreatedException: Gradle was not able to complete device setup for: dev34_aosp_atd_x86_64_Small_Phone
The emulator failed to open the managed device to generate the snapshot.
This is because the emulator closed unexpectedly (exit value = 1).

This occurred both on ubuntu-latest and after pinning to ubuntu-22.04. In all cases, the failure happens before our tests actually start running; it is strictly in the emulator / managed-device setup step.

Some example runs (URLs from the investigation):

• Last known good Android cibuildwheel run: CIBW invocation 553 (Nov 13, 2025)
• First failing run: CIBW invocation 554 (Nov 14, 2025)
• Recent failing run before tests were disabled: CIBW invocation 570 (Nov 28, 2025)

(We can fill in the exact run/job URLs when creating the issue.)

Current temporary workaround

To avoid blocking CI while still building Android wheels, we:

• Kept the Android cibuildwheel job, but
• Disabled tests via CIBW_TEST_COMMAND: "", so no emulator/Gradle tests are executed.

This means:

• We still exercise the Android build/packaging path (wheels are built for Android),
• But we no longer run the pybind11_tests test suite on Android in CI.

What we’d like help with

If you’re familiar with Android emulator / Gradle Managed Devices on GitHub Actions, help would be very welcome to:

• Identify whether this is a known issue with:
  • the Android emulator,
  • the AOSP ATD system image we’re using (API 34 x86_64), or
  • the ubuntu-24.04 runner image / KVM configuration.
• Suggest a more robust configuration for cibuildwheel’s Android testbed, e.g.:
  • different managed-device settings,
  • using a non-managed device configuration,
  • additional flags or environment variables to make snapshot creation reliable.
• Propose a fix that allows us to re-enable Android tests in CI without introducing frequent flakiness.

Acceptance criteria

• Android cibuildwheel job again runs the pybind11_tests suite on at least one supported Ubuntu runner image, using an Android emulator.
• The job is stable enough to be left enabled in normal CI runs (occasional infra hiccups are fine, but it should not fail nearly every time).
• Once a robust configuration is found, we can:
  • Remove CIBW_TEST_COMMAND: "" from .github/workflows/tests-cibw.yml, and
  • Remove or update any warnings indicating that Android tests are disabled.

Until then, we will keep the current degraded mode (build-only Android wheels) and treat this issue as the single source of truth for the status of Android testing in CI.

[rwgk]

EDIT: THIS WAS FIXED WITH PR #5915

---

I'm realizing only now that the tests are disabled in three jobs, not just the one that failed (Android wheel ubuntu-latest):

[rwgk]

[henryiii]

@mhsmith Do you have any quick ideas here? Is it something that needs improved cross-compilation support, which I probably still need to review, in scikit-build-core, #5829, etc?

[mhsmith]

No, it's something lower-level to do with starting the emulator. We've started getting identical errors in the CPython CI within the last 24 hours. Meanwhile, in the Briefcase repository, we're getting a more obvious error message about running out of disk space, which I'm fixing in beeware/.github#271 based on something I previously put into cibuildwheel's own CI. You might want to try the same.

If all of these failures are actually caused by disk space, then unfortunately they'll affect all cibuildwheel Android users, especially those whose CI jobs consume a lot of disk space before running the emulator. In that case, cibuildwheel (or the underlying test script from CPython) will need to at least do a pre-emptive disk space check to give a useful error message, and ideally free up some space automatically by itself.

For example, it should be safe enough to remove all NDK versions except the one we're currently using. There are 4 versions pre-installed on the runner, and they're about 2.5 GB each. Like the emulator hardware acceleration config, we would only do this when running within CI.

[henryiii]

@rwgk Could you try it?

        docker system prune -a -f
        sudo rm -rf $ANDROID_HOME/ndk /opt/hostedtoolcache/CodeQL \
          /usr/local/lib/node_modules /usr/local/share/chromium \
          /usr/local/share/powershell
        df -h

This PR doesn't seem to be reenabling anything, but deleting stuff instead, or I would have tried it.

[henryiii]

Oh, no, it's deleting a skip of the test run, got it. So it's currently working? Maybe adding a clean command like above would make it more reliable, though?

[rwgk]

So it's currently working?

The last commit here has [skip ci].

I'll merge master, then we'll see.

[rwgk]

Still failing.

Could you try it?

I'm super busy today and will be even more busy tomorrow.

Please go ahead and push commits here. (I was actually hoping someone else will take over.)

[mhsmith]

I've confirmed in freakboy3742/pyspamsum#108 that this error is indeed caused by disk space. There are some ideas there which will eventually find their way into cibuildwheel.

Meanwhile, I suggest you try this workaround from cibuildwheel's own CI, which is slightly different from the script above because it doesn't delete the version of the NDK which cibuildwheel is going to use.