Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

running `screenshots` with Simulator app closed #13

Closed
zanuka opened this Issue Mar 5, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@zanuka
Copy link

zanuka commented Mar 5, 2019

was able to reproduce on iMac and MacBook Air, both running XCode 10.1, MacOS 10.14.3

I could run screenshots just fine w/ the following config:

devices:
  ios:
    - iPhone X
    - iPhone 7 Plus
    - iPad Pro (12.9-inch) (2nd generation)

but only with the Simulator app opened, and all devices closed.

if I try to run with the Simulator app closed, I get driver test failures that look like this:

00:22 +11 -1: smoke test DashboardScreen > take screenshot [E]
  DriverError: Error in Flutter application: Timeout while executing waitUntilNoTransientCallbacks: TimeoutException after 0:00:05.000000: Future not completed
  #0      FlutterDriverExtension.call (package:flutter_driver/src/extension/extension.dart:180:31)
  <asynchronous suspension>
  #1      BindingBase.registerServiceExtension.<anonymous closure> (package:flutter/src/foundation/binding.dart:438:32)
  <asynchronous suspension>
  #2      _runExtension (dart:developer/runtime/libdeveloper.dart:86:23)

  Original error: null
  Original stack trace:
  null

  package:flutter_driver/src/driver/driver.dart 440:7   FlutterDriver._sendCommand
  ===== asynchronous gap ===========================
  dart:async/future_impl.dart 22:43                     _Completer.completeError
  dart:async/runtime/libasync_patch.dart 40:18          _AsyncAwaitCompleter.completeError
  package:flutter_driver/src/driver/driver.dart         FlutterDriver._sendCommand
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1053:19                          _CustomZone.registerUnaryCallback
  dart:async/runtime/libasync_patch.dart 77:23          _asyncThenWrapperHelper
  package:flutter_driver/src/driver/driver.dart         FlutterDriver._sendCommand
  package:flutter_driver/src/driver/driver.dart 490:11  FlutterDriver.waitUntilNoTransientCallbacks
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1053:19                          _CustomZone.registerUnaryCallback
  dart:async/runtime/libasync_patch.dart 77:23          _asyncThenWrapperHelper
  package:test_api/src/backend/declarer.dart            Declarer.test.<fn>.<fn>.<fn>
  package:test_api/src/backend/invoker.dart 249:15      Invoker.waitForOutstandingCallbacks.<fn>
  ===== asynchronous gap ===========================
  dart:async/zone.dart 1045:19                          _CustomZone.registerCallback
  dart:async/zone.dart 962:22                           _CustomZone.bindCallbackGuarded
  dart:async/timer.dart 52:45                           new Timer
  dart:async/timer.dart 87:9                            Timer.run
  dart:async/future.dart 174:11                         new Future
  package:test_api/src/backend/invoker.dart 398:11      Invoker._onRun.<fn>.<fn>.<fn>

running flutter driver --target=test_driver/smoke.dart from CLI

generates No devices found. which I had expected if the Simulator app is closed.

@mmcc007

This comment has been minimized.

Copy link
Owner

mmcc007 commented Mar 5, 2019

Not able to reproduce locally. But was able to start a test from the command line without running Simulator.app.

To start a test from the command line without Simulator.app running, make sure nothing is running with:

$ flutter devices
No devices detected.
$ ps ax | grep Simulator.app

and try:

xcrun simctl list devices
xcrun simctl boot <device id>

Flutter should find the device and be able to run test without Simulator.app running:

flutter devices
ps ax | grep Simulator.app
flutter driver --target=test_driver/smoke.dart

If test runs, then it should run in Screenshots.

If test reproduces this problem then it is something specific to the machine (I'm guessing).

As you mentioned, this smoke test runs on Travis on simulator?

@zanuka

This comment has been minimized.

Copy link
Author

zanuka commented Mar 5, 2019

Thanks for looking into this, it must be something on my machine / local config. It hadn't run on Travis because the app just kept failing to start. I'll spend more time creating a pipeline on a basic flutter app scaffold and putting all the pieces together, test some more.

@zanuka zanuka closed this Mar 5, 2019

@zanuka

This comment has been minimized.

Copy link
Author

zanuka commented Mar 5, 2019

@mmcc007, not sure if this helps isolate the issue, but here's the output from the sequence of steps you recommended above:

~/workspace/tierratech/logmate_flutter update-screenshots*
❯ flutter devices
1 connected device:

iPhone 7 Plus • B6BA0092-0A54-480F-87D2-ED8524D85365 • ios • iOS 12.1 (simulator)

~/workspace/tierratech/logmate_flutter update-screenshots*
❯ ps ax | grep Simulator.app
98352 s002  R+     0:00.00 grep --color=auto Simulator.app

~/workspace/tierratech/logmate_flutter update-screenshots*
❯ flutter driver --target=test_driver/smoke.dart
Using device iPhone 7 Plus.
Starting application: test_driver/smoke.dart
Running Xcode build...                                                  
 ├─Assembling Flutter resources...                           1.9s
 └─Compiling, linking and signing...                         6.5s
Xcode build done.                                           10.3s
flutter: Observatory listening on http://127.0.0.1:55664/
00:00 +0: smoke test (setUpAll)
[info ] FlutterDriver: Connecting to Flutter application at http://127.0.0.1:55664/
[trace] FlutterDriver: Isolate found with number: 804140175
[trace] FlutterDriver: Isolate is paused at start.
[trace] FlutterDriver: Attempting to resume isolate
[trace] FlutterDriver: Waiting for service extension
[info ] FlutterDriver: >>> {command: get_health}
[Crashlytics] Version 3.12.0 (136)
[info ] FlutterDriver: <<< {isError: false, response: {status: ok}, type: _extensionType, method: ext.flutter.driver}
[info ] FlutterDriver: Connected to Flutter application.
00:00 +0: smoke test SplashScreen > verify signup button is rendered
[info ] FlutterDriver: >>> {command: waitFor, finderType: ByText, text: SIGN UP}
flutter: FINEST AuthService getLocalAuthMethod methodName: null
flutter: FINEST AuthService getLocalAuthMethod localAuthMethod: LocalAuthMethod.none
flutter: FINEST AppState App.build builder: ConnectionState.done
flutter: FINEST AppState App.build snapshot: true LocalAuthMethod.none
flutter: FINEST AppState App.build isAuthenticated: false
[info ] FlutterDriver: <<< {isError: false, response: {}, type: _extensionType, method: ext.flutter.driver}
00:01 +1: smoke test SplashScreen > verify divider text is rendered
[info ] FlutterDriver: >>> {command: waitFor, finderType: ByText, text: - or -}
[info ] FlutterDriver: <<< {isError: false, response: {}, type: _extensionType, method: ext.flutter.driver}
00:01 +2: smoke test SplashScreen > verify login button is rendered
[info ] FlutterDriver: >>> {command: waitFor, finderType: ByText, text: LOG IN}
[info ] FlutterDriver: <<< {isError: false, response: {}, type: _extensionType, method: ext.flutter.driver}
00:01 +3: smoke test SplashScreen > take screenshot
[info ] FlutterDriver: >>> {command: waitUntilNoTransientCallbacks}
[info ] FlutterDriver: <<< {isError: false, response: null, type: _extensionType, method: ext.flutter.driver}
flutter: FINEST SqlfliteDataProviderService fetchTokens DB.isOpen true
flutter: FINEST SqlfliteDataProviderService fetchTokens records: []
        [C1.1 03374E72-C722-4B9F-8127-82C27F34617D 192.168.1.151:55667<->50.17.205.178:443]
        Connected Path: satisfied (Path is satisfied), interface: en0
        Duration: 2.018s, DNS @0.002s took 0.004s, TCP @0.008s took 0.189s, TLS took 0.395s
        bytes in/out: 7270/1320, packets in/out: 7/3, rtt: 0.191s, retransmitted packets: 0, out-of-order packets: 0
        [C3.1 3AF3B8F9-5A2C-4CE2-93A3-665609FD8246 192.168.1.151:55669<->23.23.73.75:443]
        Connected Path: satisfied (Path is satisfied), interface: en0
        Duration: 1.158s, DNS @0.000s took 0.003s, TCP @0.006s took 0.192s, TLS took 0.399s
        bytes in/out: 6386/8498, packets in/out: 7/9, rtt: 0.189s, retransmitted packets: 0, out-of-order packets: 0
Screenshot SplashScreen created
00:03 +4: smoke test LoginScreen > verify login button is rendered
[info ] FlutterDriver: >>> {command: tap, finderType: ByText, text: LOG IN}
[info ] FlutterDriver: <<< {isError: false, response: {}, type: _extensionType, method: ext.flutter.driver}
[info ] FlutterDriver: >>> {command: waitFor, finderType: ByText, text: LOGIN}
@zanuka

This comment has been minimized.

Copy link
Author

zanuka commented Mar 5, 2019

and these appear after every test:

01:20 +11 -2: smoke test LogbookScreen > take screenshot
[info ] FlutterDriver: >>> {command: waitUntilNoTransientCallbacks}
[warning] FlutterDriver: waitUntilNoTransientCallbacks message is taking a long time to complete...

note: this is with the Simulator.app closed, they all work when it is open. curious what version of XCode you are running.

❯ screenshots
Warning: multiple versions of 'iPhone 7 Plus' detected.
  Using 'iPhone 7 Plus' with iOS: iOS 12.1, ID: B6BA0092-0A54-480F-87D2-ED8524D85365.
Clearing images in ios/fastlane/screenshots/en-AU for 'iPhone 7 Plus'...
Starting simulator 'iPhone 7 Plus' in locale en-AU ...
Capturing screenshots with test test_driver/smoke.dart on simulator iPhone 7 Plus in locale en-AU ...
Using device iPhone 7 Plus.
Starting application: test_driver/smoke.dart
@zanuka

This comment has been minimized.

Copy link
Author

zanuka commented Mar 5, 2019

figured out the issue --- the driver tests fail if I don't manually click simulator permissions that appear on certain tests like 'Always Allow' location and 'Accept Notifications'

so we're pulling the plug on screenshots Travis integration for now until new strategy on smoke tests are decided upon

@mmcc007

This comment has been minimized.

Copy link
Owner

mmcc007 commented Mar 5, 2019

OK, ... interested in how it turns out...

BTW: it's possible to configure the timeout on waitUntilNoTransientCallbacks
(I added a demo to #14 see: https://travis-ci.org/mmcc007/screenshots/builds/502300580 for travis run )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.