Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Widget test with await is hanged when not using runAsync instead of showing error/warning message #5728

Open
drewwarren opened this issue Sep 2, 2016 · 13 comments
Labels
a: error message Error messages from the Flutter framework a: quality A truly polished experience. a: tests "flutter test", flutter_test, or one of our tests. found in release: 1.21 Found to occur in 1.21 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on. P5 Priority 5 issue (default for new feature requests; things we'd like to work on)

Comments

@drewwarren
Copy link

drewwarren commented Sep 2, 2016

import "dart:async";

import "package:flutter_test/flutter_test.dart";

void main() {
  Future future;

  setUp(() {
    future = new Future.value();
  });

  testWidgets("awaiting future with value from setUp works",
      (WidgetTester tester) async {
    await future;
  });
}
flutter_test_async$ flutter test test/async_widget_test.dart 
00:05 +0 -1: - awaiting future with value from setUp works                                         
  TimeoutException after 0:00:05.000000: Test timed out after 5 seconds.
  dart:async                _StreamController.add
  websocket_impl.dart 1111  _WebSocketImpl._WebSocketImpl._fromSocket.<fn>
  dart:async                _EventSinkWrapper.add
  websocket_impl.dart 333   _WebSocketProtocolTransformer._messageFrameEnd
  websocket_impl.dart 228   _WebSocketProtocolTransformer.add

00:05 +0 -1: Some tests failed.        
[✓] Flutter (on Mac OS, channel unknown)
    • Flutter at /Users/drewwarren/flutter
    • Framework revision 852a00a17b (8 days ago), 2016-08-25 14:23:13
    • Engine revision c4022b61fa
    • Tools Dart version 1.19.0-dev.5.0

[x] Android toolchain - develop for Android devices
    x Android Studio / Android SDK not found. Download from https://developer.android.com/sdk/
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[✓] iOS toolchain - develop for iOS devices (Xcode 7.3.1)
    • XCode at /Applications/Xcode.app/Contents/Developer
    • Xcode 7.3.1, Build version 7D1014

[✓] Atom - a lightweight development environment for Flutter
    • flutter plugin version 0.2.4
    • dartlang plugin version 0.6.37
@Hixie
Copy link
Member

Hixie commented Sep 2, 2016

testWidgets is using a FakeAsync, and setUp's future is created outside that FakeAsync scope.
I tried reproducing this using only FakeAsync, but couldn't, so I'm not 100% sure what's going on.
@yjbanov any ideas before I try to minimise the heck out of flutter_test? :-)

@yjbanov
Copy link
Contributor

yjbanov commented Sep 3, 2016

This does look zone-related. I couldn't reproduce using just FakeAsync either. However, what I noticed is that in a plain package:test + FakeAsync test the test zone is a direct child of the setUp zone, but it is a grandchild in a package:flutter_test test.

// package:test
setUp zones: 421730508
test zones: 180409779 < 421730508
// package:flutter_test
setUp zones: 585864271
test zones: 799539283 < 436113664 < 585864271

Here's the plain code I used:

import "dart:async";

import "package:quiver/testing/async.dart";
import "package:test/test.dart";

void main() {
  Future future;

  zoneChain() {
    var zone = Zone.current;
    var zones = [];
    while(zone != null) {
      zones.add(zone);
      zone = zone.parent;
    }
    return zones.map((z) => z.hashCode).join(' < ');
  }

  setUp(() {
    print('setUp zones: ${zoneChain()}');
    future = new Future.value();
  });

  test("awaiting future with value from setUp works", () async {
    var fakeAsync = new FakeAsync();
    fakeAsync.run((_) async {
      print('test zones: ${zoneChain()}');
      await future;
    });
    fakeAsync.flushMicrotasks();
  });
}

@Hixie
Copy link
Member

Hixie commented Sep 3, 2016

@nex3 any ideas?

@Hixie Hixie modified the milestone: Flutter 1.0 Sep 12, 2016
@Hixie Hixie added the framework flutter/packages/flutter repository. See also f: labels. label Sep 12, 2016
@eseidelGoogle
Copy link
Contributor

Do we think this is a package:test bug, should we move it there?

@Hixie
Copy link
Member

Hixie commented Feb 3, 2017

we've no idea.

@zoechi
Copy link
Contributor

zoechi commented Aug 27, 2018

Is this still an issue?

@apaatsio
Copy link
Contributor

apaatsio commented Oct 22, 2018

From what I understand you are supposed to use tester.runAsync() when you want to test async code like that with testWidgets.

This works:

testWidgets("awaiting future with value from setUp works",
    (WidgetTester tester) async {
  await tester.runAsync(() async {
    await future;
  });
});

Edit: added await to tester.runAsync().

@yjbanov
Copy link
Contributor

yjbanov commented Oct 22, 2018

I believe @mdebbar saw this recently too. Yes, any real asynchronous work has to go through tester.runAsync(). However, we should try to find a way for tests to not hang, but instead fail with a message complaining about real async work done inside the test. Not sure if that's possible 🤔

@mdebbar
Copy link
Contributor

mdebbar commented Oct 22, 2018

And don’t forget to await on runAsync:

await tester.runAsync(() async {
  await future;
});

@goderbauer goderbauer added the a: tests "flutter test", flutter_test, or one of our tests. label Dec 27, 2018
@Hixie Hixie removed their assignment Jul 16, 2019
@Hixie Hixie modified the milestones: Stretch Goals, New Stretch Goals Jan 7, 2020
@kf6gpe kf6gpe added the P5 Priority 5 issue (default for new feature requests; things we'd like to work on) label May 29, 2020
@kf6gpe kf6gpe modified the milestone: Stretch Goals Jun 1, 2020
@Hixie Hixie removed this from the [DEPRECATED] Stretch Goals milestone Jun 16, 2020
@kf6gpe kf6gpe removed this from the [DEPRECATED] Stretch Goals milestone Jul 7, 2020
@TahaTesser
Copy link
Member

Based on #5728 (comment), test still get hanged when not runAsync so keeping the issue open but #5728 (comment) solves the issue

Code Sample
import "dart:async";

import "package:flutter_test/flutter_test.dart";

void main() {
  Future future;

  setUp(() {
    future = new Future.value();
  });

  testWidgets("awaiting future with value from setUp works",
      (WidgetTester tester) async {
    await tester.runAsync(() async {
      await future;
    });
  });
}

flutter doctor -v
[✓] Flutter (Channel dev, 1.21.0-1.0.pre, on Mac OS X 10.15.6 19G73, locale en-GB)
    • Flutter version 1.21.0-1.0.pre at /Users/taha/Code/flutter_dev
    • Framework revision f25bd9c55c (6 days ago), 2020-07-14 20:26:01 -0400
    • Engine revision 99c2b3a245
    • Dart version 2.9.0 (build 2.9.0-21.0.dev 20bf2fcf56)

 
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/taha/Code/sdk
    • Platform android-29, build-tools 29.0.3
    • ANDROID_HOME = /Users/taha/Code/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.6)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.6, Build version 11E708
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 47.1.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.47.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.12.2

[✓] Connected device (5 available)
    • SM M305F (mobile)      • 32003c30dc19668f          • android-arm64  • Android 10 (API 29)
    • Taha’s iPhone (mobile) • 00008020-001059882212002E • ios            • iOS 13.6
    • macOS (desktop)        • macos                     • darwin-x64     • Mac OS X 10.15.6 19G73
    • Web Server (web)       • web-server                • web-javascript • Flutter Tools
    • Chrome (web)           • chrome                    • web-javascript • Google Chrome 84.0.4147.89

• No issues found!

@TahaTesser TahaTesser added found in release: 1.21 Found to occur in 1.21 has reproducible steps The issue has been confirmed reproducible and is ready to work on. labels Jul 21, 2020
@TahaTesser TahaTesser changed the title Widget tests timeout if you await a future defined in setUp Widget test with await is hanged when not using runAsync instead of showing error/warning message Jul 21, 2020
@TahaTesser TahaTesser added the a: error message Error messages from the Flutter framework label Jul 21, 2020
@TahaTesser TahaTesser added the a: quality A truly polished experience. label Jul 21, 2020
@kf6gpe kf6gpe modified the milestone: [DEPRECATED] Stretch Goals Jul 22, 2020
@Hixie Hixie removed this from the - milestone Aug 17, 2020
@Jonas-Sander
Copy link

So it is on purpose that Future.delayed(Duration.zero) or pumpEventQueue() both will just time out?

I came here because I wanted to extract creating a bloc (and closing it) inside a test into a setUp method:

Old code:

      testWidgets('placeholder is shown on open homework page when no homeworks are to do',
          (tester) async {
        // Creating it here
        final homeworkPageBloc = MockTeacherHomeworkPageBloc();

        await _pumpHomeworkPageWithPlaceholder(tester,
            bloc: homeworkPageBloc, initialTab: HomeworkTab.open);

        expect(
            find.byKey(
                ValueKey('no-homework-teacher-placeholder-for-open-homework')),
            findsOneWidget);

        homeworkPageBloc.close();
      });

New code:

      setUp(() {
        // Creating it here
        homeworkPageBloc = MockTeacherHomeworkPageBloc();
      });

      testWidgets('placeholder is shown on open homework page when no homeworks are to do',
          (tester) async {
        await pumpHomeworkPage(tester,
            bloc: homeworkPageBloc, initialTab: HomeworkTab.open);

        homeworkPageBloc.emitNewState(_noHomeworks);

        await tester.pump();

        expect(
            find.byKey(
                ValueKey('no-homework-teacher-placeholder-for-open-homework')),
            findsOneWidget);

        homeworkPageBloc.close();
      });

But when I do this the test will fail.
It fails because in the test I use homeworkPageBloc.emitNewState(_noHomeworks); which tells the bloc to emit my needed state.
When I use this method with the bloc created inside the test it works (the new state is emitted).
If I create the Bloc inside the setUp method it won't work anymore, the wanted state is only emitted after the test has already executed.
I doesn't matter how often I call pump. I can't call pumpAndSettle because of an ever-lasting animation and pumpEventQueue does not work in the widget test.

I already wasted so much time because this behavior is non-obvious. Especially if you have complexity through something like a bloc. As a normal flutter user you just don't know all this behavior. I still don't understand why this test fails exactly.

bloc.emitNewState is a method on my mock bloc which is still implemented kinda janky:

class MockTeacherHomeworkPageBloc extends HomeworkPageBloc {
  final _queuedStates = Queue<HomeworkPageState>();

  void emitNewState(HomeworkPageState state) {
    _queuedStates.add(state);
    add(LoadHomeworks());
  }

  @override
  Stream<HomeworkPageState> mapEventToState(HomeworkPageEvent event) async* {
    if (_queuedStates.isNotEmpty) {
      yield _queuedStates.removeFirst();
    }
  }

  @override
  HomeworkPageState initialState = Loading();
}

In emitNewState add(LoadHomeworks()); is only called so that mapEventToState is eventually called by the bloc framework (asynchronously - thats probably a problem?).
It's a bit of a hack but I still expected this to go smoothly especially because the call of this method has nothing to do with the bloc being set up in the setUp method or in the test body.

@maks
Copy link

maks commented Apr 7, 2021

@Jonas-Sander I think this is maybe catching out new Flutter developers because the way that widget tests run inside zones created by FakeAsync and then how FakeAsync works with streams is not covered at all in the widget testing intro documentation.

BUT I think you should raise that as a new issue as that's not really what this issue is about.

@Jonas-Sander
Copy link

is not covered at all in the widget testing intro documentation.

And even this assumes that everyone reads the docs.

BUT I think you should raise that as a new issue as that's not really what this issue is about.

What would the premise of the new issue be exactly? The docs? The behavior overall?

Do you have a recommendation for my case? Just skipping the setUp? Using integration_test (won't this take much longer then?) Not using Bloc? This is really confusing behavior :/
(For now I'm just gonna skip creating the bloc in setUp but I'm a bit scared for other problems with blocs and testWidgets)

dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Oct 7, 2022
- Change `testWidgets` to `test` to improve async handling (`testWidgets` uses FakeAsync: flutter/flutter#5728)
- Fix newly-failing tests for `next`

commit-id:0d79ab75
dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Oct 7, 2022
- Change `testWidgets` to `test` to improve async handling (`testWidgets` uses FakeAsync: flutter/flutter#5728)
- Fix newly-failing tests for `next`

commit-id:0d79ab75
dnys1 added a commit to dnys1/amplify-flutter that referenced this issue Oct 7, 2022
- Change `testWidgets` to `test` to improve async handling (`testWidgets` uses FakeAsync: flutter/flutter#5728)
- Fix newly-failing tests for `next`

commit-id:0d79ab75
dnys1 added a commit to aws-amplify/amplify-flutter that referenced this issue Oct 11, 2022
- Change `testWidgets` to `test` to improve async handling (`testWidgets` uses FakeAsync: flutter/flutter#5728)
- Fix newly-failing tests for `next`

commit-id:0d79ab75
dnys1 added a commit to aws-amplify/amplify-flutter that referenced this issue Oct 12, 2022
- Change `testWidgets` to `test` to improve async handling (`testWidgets` uses FakeAsync: flutter/flutter#5728)
- Fix newly-failing tests for `next`

commit-id:0d79ab75
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a: error message Error messages from the Flutter framework a: quality A truly polished experience. a: tests "flutter test", flutter_test, or one of our tests. found in release: 1.21 Found to occur in 1.21 framework flutter/packages/flutter repository. See also f: labels. has reproducible steps The issue has been confirmed reproducible and is ready to work on. P5 Priority 5 issue (default for new feature requests; things we'd like to work on)
Projects
None yet
Development

No branches or pull requests