Tests using `integration_test` have incorrect duration when run natively #117386

bartekpacia · 2022-12-20T11:55:46Z

Steps to reproduce

Clone the bartekpacia/flutter_integration_test_duration_bug repository. This is a default counter app slightly modified by me to demonstrate this problem. cd to the cloned repository.
$ flutter pub get
$ cd android

Run Flutter integration test natively:

$ ./gradlew app:connectedDebugAndroidTest -Ptarget=$(pwd)/../integration_test/app_test.dart

Open the test summary generated by Gradle:

$ open ../build/app/reports/androidTests/connected/index.html

The same problem occurs when running tests on iOS. Refer to integration_test's README to learn how to run tests on iOS.

Expected results:

I expect the test duration to be correct.

The integration_test/app_test.dart that was run calls tester.pump(Duration(seconds: 3)); twice, so the duration should not be shorter than at least 6 seconds.

Actual results:

Test duration is very short, on my machine it usually revolves around ~300ms.

More details

This problem is caused by how Dart test results are submitted to the native side, and how native test cases are created from these Dart test results.

Currently IntegrationTestWidgetsFlutterBinding submits Dart test results to the native side after all Dart tests have been executed. When the native side receives these results, it creates native test cases (Android implementation, iOS implementation). But these test cases are simply stubs – their execution is finished immediately after creation. That's why the runtime of these testcases is so very short.

A solution that comes to my mind would do the following:

Collect all declared groups and tests declared in Dart.
Submit declared groups and tests to the native side.
The native side receives test results. This is where the test duration on native starts.
Dart tests execute, and finally finish.
Submit Dart test results to the native side

Logs

$ ./gradlew app:connectedDebugAndroidTest -Ptarget=$(pwd)/../integration_test/app_test.dart

> Task :app:compressDebugAssets
Execution optimizations have been disabled for task ':app:compressDebugAssets' to ensure correctness due to the following reasons:
  - Gradle detected a problem with the following location: '/Users/bartek/dev/random/flutter_integration_test_duration_bug/build/app/intermediates/assets/debug/mergeDebugAssets'. Reason: Task ':app:compressDebugAssets' uses this output of task ':app:copyFlutterAssetsDebug' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.4/userguide/validation_problems.html#implicit_dependency for more details about this problem.

> Task :app:connectedDebugAndroidTest
Starting 1 tests on Pixel_4_API_31(AVD) - 12

Test results saved as file:/Users/bartek/dev/random/flutter_integration_test_duration_bug/build/app/outputs/androidTest-results/connected/Pixel_4_API_31(AVD)%20-%2012/test-result.pb. Inspect these results in Android Studio by selecting Run > Import Tests From File from the menu bar and importing test-result.pb.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.

BUILD SUCCESSFUL in 16s
76 actionable tasks: 5 executed, 71 up-to-date

$ flutter doctor -v
[✓] Flutter (Channel stable, 3.3.10, on macOS 13.0.1 22A400 darwin-arm, locale en-PL)
    • Flutter version 3.3.10 on channel stable at /Users/bartek/fvm/versions/stable
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 135454af32 (5 days ago), 2022-12-15 07:36:55 -0800
    • Engine revision 3316dd8728
    • Dart version 2.18.6
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
    • Android SDK at /Users/bartek/androidsdk
    • Platform android-33, build-tools 33.0.1
    • ANDROID_HOME = /Users/bartek/androidsdk
    • Java binary at: /Users/bartek/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/213.7172.25.2113.9123335/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 14C18
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Users/bartek/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/213.7172.25.2113.9123335/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Ultimate Edition (version 2022.3)
    • IntelliJ at /Users/bartek/Applications/JetBrains Toolbox/IntelliJ IDEA Ultimate.app
    • Flutter plugin version 71.1.6
    • Dart plugin version 223.7571.203

[✓] IntelliJ IDEA Ultimate Edition (version 2022.3)
    • IntelliJ at /Users/bartek/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-0/223.7571.182/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.74.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.54.0

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 12 (API 31) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 13.0.1 22A400 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 108.0.5359.124

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

The text was updated successfully, but these errors were encountered:

darshankawar · 2022-12-20T13:01:57Z

Thanks for the detailed report @bartekpacia
Using the demo repo and running as mentioned, I do see the test duration is shorter (0.358 ms) in my case.

stable, master flutter doctor -v

[✓] Flutter (Channel stable, 3.3.10, on macOS 12.2.1 21D62 darwin-x64, locale
    en-GB)
    • Flutter version 3.3.10 on channel stable at
      /Users/dhs/documents/fluttersdk/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 135454af32 (15 hours ago), 2022-12-15 07:36:55 -0800
    • Engine revision 3316dd8728
    • Dart version 2.18.6
    • DevTools version 2.15.0

[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

[!] Flutter (Channel master, 3.7.0-10.0.pre.61, on macOS 12.2.1 21D62
    darwin-x64, locale en-GB)
    • Flutter version 3.7.0-10.0.pre.61 on channel master at
      /Users/dhs/documents/fluttersdk/flutter
    ! Warning: `flutter` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/flutter, which is not inside
      your current Flutter SDK checkout at
      /Users/dhs/documents/fluttersdk/flutter. Consider adding
      /Users/dhs/documents/fluttersdk/flutter/bin to the front of your path.
    ! Warning: `dart` on your path resolves to
      /Users/dhs/Documents/Fluttersdk/flutter/bin/dart, which is not inside your
      current Flutter SDK checkout at /Users/dhs/documents/fluttersdk/flutter.
      Consider adding /Users/dhs/documents/fluttersdk/flutter/bin to the front
      of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 50a23d962c (66 minutes ago), 2022-12-19 22:45:18 -0500
    • Engine revision 339791f190
    • Dart version 3.0.0 (build 3.0.0-35.0.dev)
    • DevTools version 2.20.0
    • If those were intentional, you can disregard the above warnings; however
      it is recommended to use "git" directly to perform update checks and
      upgrades.


[!] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    ! Flutter recommends a minimum Xcode version of 13.
      Download the latest version or update via the Mac App Store.
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] VS Code (version 1.62.0)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0

[✓] Connected device (5 available)
    • SM G975F (mobile)       • RZ8M802WY0X • android-arm64   • Android 11 (API 30)
    • Darshan's iphone (mobile)  • 21150b119064aecc249dfcfe05e259197461ce23 •
      ios            • iOS 14.4.1 18D61
    • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729     •
      ios            • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator)
    • macOS (desktop)            • macos                                    •
      darwin-x64     • Mac OS X 10.15.4 19E2269 darwin-x64
    • Chrome (web)               • chrome                                   •
      web-javascript • Google Chrome 98.0.4758.80

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Keeping this issue open for team's attention and insights on expected behavior.

bartekpacia mentioned this issue Dec 20, 2022

Dart test groups are removed when running tests with integration_test natively #117389

Open

darshankawar added the in triage Presently being triaged by the triage team label Dec 20, 2022

goderbauer added the P2 Important issues not at the top of the work list label Dec 20, 2022

bartekpacia mentioned this issue Feb 11, 2023

Building all integration test targets into a single application binary #115751

Open

flutter-triage-bot bot added team-framework Owned by Framework team triaged-framework Triaged by Framework team labels Jul 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests using `integration_test` have incorrect duration when run natively #117386

Tests using `integration_test` have incorrect duration when run natively #117386

bartekpacia commented Dec 20, 2022 •

edited

darshankawar commented Dec 20, 2022

Tests using integration_test have incorrect duration when run natively #117386

Tests using integration_test have incorrect duration when run natively #117386

Comments

bartekpacia commented Dec 20, 2022 • edited

Steps to reproduce

darshankawar commented Dec 20, 2022

Tests using `integration_test` have incorrect duration when run natively #117386

Tests using `integration_test` have incorrect duration when run natively #117386

bartekpacia commented Dec 20, 2022 •

edited