Linux Android Scenarios builder is failing on LUCI

[jason-simmons]

The "Linux Android Scenarios" builder on LUCI has been failing consistently. It times out after 50 minutes.

The log ends with:

dev.flutter.scenarios.EngineLaunchE2ETest > initializationError[android_23_google_apis_x86(AVD) - 6.0] FAILED
	java.lang.NoClassDefFoundError: java.util.concurrent.CompletableFuture
	at libcore.reflect.InternalNames.getClass(InternalNames.java:55)

Example: https://ci.chromium.org/p/flutter/builders/try/Linux%20Android%20Scenarios/12592?

@dnfield

[dnfield]

Looking

[dnfield]

CompletableFuture was added in API 24. The test is trying to run with API 23. @blasten fYI

[dnfield]

Cause of this was https://flutter-review.googlesource.com/c/recipes/+/15340 - @GaryQian @blasten

[CaseyHillers]

Reopening as the luci script was hiding the build failures

[jason-simmons]

There are multiple issues affecting the scenario test:

• The test fails when LUCI runs it on an API 23 emulator.
  • In that environment EngineLaunchE2ETest.java fails with a timed out waiting for engine started signal error
• When the test fails, it hangs for 50 minutes until the LUCI task timeout expires.
  • The test process on the emulator is blocked in the flutterUiRenderedLock.wait() call in TestableFlutterActivity.waitUntilFlutterRendered. That call should probably have a timeout.
• LUCI will show the test as green even if it does not build properly.
  • The LUCI recipe is ignoring the exit code of the script that builds and runs the test (https://flutter.googlesource.com/recipes/+/refs/heads/master/recipes/engine/scenarios.py#110)

[blasten]

LUCI will show the test as green even if it does not build properly.

This needs to be fixed. One issue is that we need support from Skia Gold in the engine.

There's substantial noise in Skia rolls that makes this test very flaky due to small sub-pixel variations.

[jason-simmons]

The scenario test is still not running in the API 30 emulator on LUCI even after fixing the compilation error in EngineLaunchE2ETest.java.

The test fails with an error like this:

> Task :app:connectedDebugAndroidTest
Starting 0 tests on android_30_google_apis_x86(AVD) - 11
Tests on android_30_google_apis_x86(AVD) - 11 failed: Instrumentation run failed due to 'Process crashed.'

com.android.build.gradle.internal.testing.ConnectedDevice > No tests found.[android_30_google_apis_x86(AVD) - FAILED
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack @Test annotations).

This log was taken from a CI run in flutter/engine#27414 (see https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket.appspot.com/8841614427359704064/+/u/Scenario_App_Integration_Tests/stdout)

[blasten]

@jason-simmons I will be working on moving these tests to the framework, so they won't be in the engine scenario app anymore.

For now, let's keep the task as flaky. I will close this issue once the tests have been removed from the engine. See #83856 for more.

@dnfield couldn't run these tests locally due to other issues.

[blasten]

Closing in favor of #90401

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.