Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures in integration tests are not reported correctly in Firebase Test Lab #108958

Open
ghost opened this issue Aug 4, 2022 · 3 comments
Open

Failures in integration tests are not reported correctly in Firebase Test Lab #108958

ghost opened this issue Aug 4, 2022 · 3 comments
Labels
a: tests "flutter test", flutter_test, or one of our tests f: integration_test The flutter/packages/integration_test plugin found in release: 3.0 Found to occur in 3.0 found in release: 3.1 Found to occur in 3.1 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on P3 Issues that are less important to the Flutter project team-framework Owned by Framework team triaged-framework Triaged by Framework team

Comments

@ghost
Copy link

ghost commented Aug 4, 2022

Steps to Reproduce

  1. Clone this repository: https://github.com/robertof-wolt/ftl-issue-demo
  2. Build the application according to the instruction from README.md
  3. Run the test in Firebase Test Lab according to the instructions from README.md

Expected results:
The build fails and the reports shows 1 test case, 1 failed, 0 passed

Actual results:
The build fails but the report shows 0 test cases, 0 failed, 0 passed
Screenshot 2022-08-04 at 12 55 26

Code sample See the repository at https://github.com/robertof-wolt/ftl-issue-demo
Logs 
Analyzing ftl-issue-demo...
No issues found! (ran in 8.6s)

[✓] Flutter (Channel stable, 3.0.5, on macOS 12.5 21G72 darwin-x64, locale en-FI)
    • Flutter version 3.0.5 at /Users/robertofasciolo/tools/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f1875d570e (3 weeks ago), 2022-07-13 11:24:16 -0700
    • Engine revision e85ea0e79c
    • Dart version 2.17.6
    • DevTools version 2.12.2

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/robertofasciolo/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • ANDROID_SDK_ROOT = /Users/robertofasciolo/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] IntelliJ IDEA Ultimate Edition (version 2021.3.3)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] Connected device (3 available)
    • sdk gphone x86 64 (mobile) • emulator-5554 • android-x64    • Android 11 (API 30) (emulator)
    • macOS (desktop)            • macos         • darwin-x64     • macOS 12.5 21G72 darwin-x64
    • Chrome (web)               • chrome        • web-javascript • Google Chrome 103.0.5060.134

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!
@danagbemava-nc danagbemava-nc added the in triage Presently being triaged by the triage team label Aug 4, 2022
@danagbemava-nc
Copy link
Member

Reproducible using the steps outlined in the repo readme.

Leaving this for further input from the team.

screenshot

Screenshot 2022-08-04 at 12 38 42

flutter doctor -v 
[✓] Flutter (Channel stable, 3.0.5, on macOS 12.4 21F79 darwin-arm, locale en-GB)
    • Flutter version 3.0.5 at /Users/nexus/dev/sdks/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision f1875d570e (3 weeks ago), 2022-07-13 11:24:16 -0700
    • Engine revision e85ea0e79c
    • Dart version 2.17.6
    • DevTools version 2.12.2

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/nexus/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] VS Code (version 1.69.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.46.0

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 13 (API 33) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 12.4 21F79 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 104.0.5112.79

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[✓] Flutter (Channel master, 3.1.0-0.0.pre.2084, on macOS 12.4 21F79 darwin-arm, locale en-GB)
    • Flutter version 3.1.0-0.0.pre.2084 on channel master at /Users/nexus/dev/sdks/flutters
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 1b6473f45c (4 hours ago), 2022-08-04 01:14:23 -0400
    • Engine revision 7b6911a843
    • Dart version 2.19.0 (build 2.19.0-61.0.dev)
    • DevTools version 2.16.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
    • Android SDK at /Users/nexus/Library/Android/sdk
    • Platform android-33, build-tools 33.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.4.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 13F100
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840)

[✓] VS Code (version 1.69.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.46.0

[✓] Connected device (3 available)
    • sdk gphone64 arm64 (mobile) • emulator-5554 • android-arm64  • Android 13 (API 33) (emulator)
    • macOS (desktop)             • macos         • darwin-arm64   • macOS 12.4 21F79 darwin-arm
    • Chrome (web)                • chrome        • web-javascript • Google Chrome 104.0.5112.79

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

@danagbemava-nc danagbemava-nc added framework flutter/packages/flutter repository. See also f: labels. f: integration_test The flutter/packages/integration_test plugin has reproducible steps The issue has been confirmed reproducible and is ready to work on found in release: 3.0 Found to occur in 3.0 found in release: 3.1 Found to occur in 3.1 a: tests "flutter test", flutter_test, or one of our tests and removed in triage Presently being triaged by the triage team labels Aug 4, 2022
@goderbauer
Copy link
Member

cc @pdblasi-google

@goderbauer goderbauer added the P3 Issues that are less important to the Flutter project label Aug 9, 2022
@ghost
Copy link
Author

ghost commented Aug 10, 2022

The same thing happens in SauceLabs so it doesn't seem to be farm cloud specific.

@flutter-triage-bot flutter-triage-bot bot added team-framework Owned by Framework team triaged-framework Triaged by Framework team labels Jul 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
a: tests "flutter test", flutter_test, or one of our tests f: integration_test The flutter/packages/integration_test plugin found in release: 3.0 Found to occur in 3.0 found in release: 3.1 Found to occur in 3.1 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on P3 Issues that are less important to the Flutter project team-framework Owned by Framework team triaged-framework Triaged by Framework team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@goderbauer @danagbemava-nc