[integration_test] Test doesn't start when uploaded APK to Firebase Test Lab

[TahaTesser]

Continuation from #85728 (comment)

I can reproduce an issue with integration_test running on the Firebase test lab using a regular Flutter app. Following the steps as #85728 (comment)

Code Sample Repository

https://github.com/tesserx/flutter_integration_ftl
(note: add your own maps API key)

Steps to reproduce

1. Create a regular project on the latest stable, 2.2.3
2. Add integration_test dependency, add a very simple test to find app title

  testWidgets('Find App Title', (WidgetTester tester) async {
    await tester.pumpWidget(MyApp());

    expect(find.text('Material App Bar'), findsOneWidget);
  });

3. Run the test locally with flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart

4. Follow instructions to upload an APK to Firebase Test Lab

5. Upload the APK, the test passes. check out test screenshots and video, logs

Firebase test lab says tests have passed, video is only around a minute, screenshots show the test started and the app was running (can see home screen)

Now let's make it a little bit more complex, for which I simply added another screen with the google_maps_flutter example.
6. Add button to navigate to this maps page, click a button and navigate back to home screen

  testWidgets('Test Minimal Googel Maps Sample', (WidgetTester tester) async {
    await tester.pumpWidget(MyApp());

    final mapsSample = find.text('Maps Minimimal');
    expect(mapsSample, findsOneWidget);
    await tester.tap(mapsSample);
    await tester.pumpAndSettle();
    await tester.pump(new Duration(seconds: 5));

    final fab = find.byKey(Key('fabKey'));
    expect(fab, findsOneWidget);
    await tester.tap(fab);
    await tester.pumpAndSettle();
    await tester.pump(new Duration(seconds: 3));

    final back = find.byTooltip('Back');
    expect(back, findsOneWidget);
    await tester.tap(back);
    await tester.pumpAndSettle();

    expect(mapsSample, findsOneWidget);
  });

7. Repeat step 3, notice locally test passes, navigates maps screen, and navigates back every time

Screen.Recording.2021-07-09.at.4.45.25.PM.mov

8. Repeat step 4. After the test finishes, the Firebase Test lab shows tests passed
   Check out test results

• Screenshots shows the app was stuck at "Test starting..." for the entire time

preview

• Video results show the same thing, video length is 5 min (which is the default timeout for FTL), it's just 5 min of monkey test taps
• No actual integration_test was done for the entire duration of the test.

preview

web-build_2021-07-08T13_05_03.590Z_auzl_blueline-28-en_US-portrait_video.mp4

• When checking logcat from this test, there are so many error messages
  logcat.zip

Thank you

[TahaTesser]

cc @jiahaog @dnfield

[darshankawar]

Verified this on latest stable and observed same behavior, ie, the test seems to be stuck at initial setup (test starting) but doesn't actually proceed to run the test. FTL shows the test passed, but screenshots shows same screen test starting throughout the duration of test execution:

ftl.mp4

flutter doctor -v

[✓] Flutter (Channel stable, 2.2.3, on Mac OS X 10.15.4 19E2269 darwin-x64,
    locale en-GB)
    • Flutter version 2.2.3 at /Users/dhs/documents/fluttersdk/flutter
    • Framework revision f4abaa0735 (4 days ago), 2021-07-01 12:46:11 -0700
    • Engine revision 241c87ad80
    • Dart version 2.13.4

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.3, Build version 12C33
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.57.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (3 available)
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                               •
      web-javascript • Google Chrome 91.0.4472.114

• No issues found!

[gbminnock]

I also have this issue, any updates on this?

[mosofsky]

I am also experiencing this issue, however, the Firebase Test Lab video doesn't even say "Test Starting...". I just see a white screen. The test passes on my Mac but times out on Firebase Test Lab.

It's reproducible with these variants:

• Upload to Firebase Test Lab via gcloud utility AND upload through the Firebase Test Lab UI
• With screenshot code AND without screenshot code
• With emulator AND with physical device

matrix-2nre8ujlpfmo2.mp4

Any help greatly appreciated.

[mosofsky]

I was able to get Firebase Test Lab to test a Flutter App by using this Espresso "bridge" https://pub.dev/packages/espresso. So instead of writing Flutter tests with the integrationDriver from package:integration_test/integration_test_driver.dart as described in https://github.com/flutter/flutter/tree/main/packages/integration_test#driver-entrypoint, I can use the enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart.

Not ideal.

Update: Flutter's Espresso plugin causes "Duplicate class" compilation errors when added to a project that contains Flutter Firebase SDKs, e.g. cloud_firestore: ^4.1.0. I filed that issue here: #119252

[OlegNovosad]

Just to clean up this issue - it is not reproducible. I have not issues running this one on Firebase Test Lab.