Widget test with await is hanged when not using runAsync instead of showing error/warning message

[drewwarren]

import "dart:async";

import "package:flutter_test/flutter_test.dart";

void main() {
  Future future;

  setUp(() {
    future = new Future.value();
  });

  testWidgets("awaiting future with value from setUp works",
      (WidgetTester tester) async {
    await future;
  });
}

flutter_test_async$ flutter test test/async_widget_test.dart 
00:05 +0 -1: - awaiting future with value from setUp works                                         
  TimeoutException after 0:00:05.000000: Test timed out after 5 seconds.
  dart:async                _StreamController.add
  websocket_impl.dart 1111  _WebSocketImpl._WebSocketImpl._fromSocket.<fn>
  dart:async                _EventSinkWrapper.add
  websocket_impl.dart 333   _WebSocketProtocolTransformer._messageFrameEnd
  websocket_impl.dart 228   _WebSocketProtocolTransformer.add

00:05 +0 -1: Some tests failed.        

[✓] Flutter (on Mac OS, channel unknown)
    • Flutter at /Users/drewwarren/flutter
    • Framework revision 852a00a17b (8 days ago), 2016-08-25 14:23:13
    • Engine revision c4022b61fa
    • Tools Dart version 1.19.0-dev.5.0

[x] Android toolchain - develop for Android devices
    x Android Studio / Android SDK not found. Download from https://developer.android.com/sdk/
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[✓] iOS toolchain - develop for iOS devices (Xcode 7.3.1)
    • XCode at /Applications/Xcode.app/Contents/Developer
    • Xcode 7.3.1, Build version 7D1014

[✓] Atom - a lightweight development environment for Flutter
    • flutter plugin version 0.2.4
    • dartlang plugin version 0.6.37

[Hixie]

testWidgets is using a FakeAsync, and setUp's future is created outside that FakeAsync scope.
I tried reproducing this using only FakeAsync, but couldn't, so I'm not 100% sure what's going on.
@yjbanov any ideas before I try to minimise the heck out of flutter_test? :-)

[yjbanov]

This does look zone-related. I couldn't reproduce using just FakeAsync either. However, what I noticed is that in a plain package:test + FakeAsync test the test zone is a direct child of the setUp zone, but it is a grandchild in a package:flutter_test test.

// package:test
setUp zones: 421730508
test zones: 180409779 < 421730508

// package:flutter_test
setUp zones: 585864271
test zones: 799539283 < 436113664 < 585864271

Here's the plain code I used:

import "dart:async";

import "package:quiver/testing/async.dart";
import "package:test/test.dart";

void main() {
  Future future;

  zoneChain() {
    var zone = Zone.current;
    var zones = [];
    while(zone != null) {
      zones.add(zone);
      zone = zone.parent;
    }
    return zones.map((z) => z.hashCode).join(' < ');
  }

  setUp(() {
    print('setUp zones: ${zoneChain()}');
    future = new Future.value();
  });

  test("awaiting future with value from setUp works", () async {
    var fakeAsync = new FakeAsync();
    fakeAsync.run((_) async {
      print('test zones: ${zoneChain()}');
      await future;
    });
    fakeAsync.flushMicrotasks();
  });
}

[Hixie]

@nex3 any ideas?

[eseidelGoogle]

Do we think this is a package:test bug, should we move it there?

[Hixie]

we've no idea.

[zoechi]

Is this still an issue?

[apaatsio]

From what I understand you are supposed to use tester.runAsync() when you want to test async code like that with testWidgets.

This works:

testWidgets("awaiting future with value from setUp works",
    (WidgetTester tester) async {
  await tester.runAsync(() async {
    await future;
  });
});

Edit: added await to tester.runAsync().

[yjbanov]

I believe @mdebbar saw this recently too. Yes, any real asynchronous work has to go through tester.runAsync(). However, we should try to find a way for tests to not hang, but instead fail with a message complaining about real async work done inside the test. Not sure if that's possible 🤔

[mdebbar]

And don’t forget to await on runAsync:

await tester.runAsync(() async {
  await future;
});

[TahaTesser]

Based on #5728 (comment), test still get hanged when not runAsync so keeping the issue open but #5728 (comment) solves the issue

Code Sample

import "dart:async";

import "package:flutter_test/flutter_test.dart";

void main() {
  Future future;

  setUp(() {
    future = new Future.value();
  });

  testWidgets("awaiting future with value from setUp works",
      (WidgetTester tester) async {
    await tester.runAsync(() async {
      await future;
    });
  });
}

flutter doctor -v

[✓] Flutter (Channel dev, 1.21.0-1.0.pre, on Mac OS X 10.15.6 19G73, locale en-GB)
    • Flutter version 1.21.0-1.0.pre at /Users/taha/Code/flutter_dev
    • Framework revision f25bd9c55c (6 days ago), 2020-07-14 20:26:01 -0400
    • Engine revision 99c2b3a245
    • Dart version 2.9.0 (build 2.9.0-21.0.dev 20bf2fcf56)

 
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/taha/Code/sdk
    • Platform android-29, build-tools 29.0.3
    • ANDROID_HOME = /Users/taha/Code/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.6)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.6, Build version 11E708
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 47.1.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.47.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.12.2

[✓] Connected device (5 available)
    • SM M305F (mobile)      • 32003c30dc19668f          • android-arm64  • Android 10 (API 29)
    • Taha’s iPhone (mobile) • 00008020-001059882212002E • ios            • iOS 13.6
    • macOS (desktop)        • macos                     • darwin-x64     • Mac OS X 10.15.6 19G73
    • Web Server (web)       • web-server                • web-javascript • Flutter Tools
    • Chrome (web)           • chrome                    • web-javascript • Google Chrome 84.0.4147.89

• No issues found!

[Jonas-Sander]

So it is on purpose that Future.delayed(Duration.zero) or pumpEventQueue() both will just time out?

I came here because I wanted to extract creating a bloc (and closing it) inside a test into a setUp method:

Old code:

      testWidgets('placeholder is shown on open homework page when no homeworks are to do',
          (tester) async {
        // Creating it here
        final homeworkPageBloc = MockTeacherHomeworkPageBloc();

        await _pumpHomeworkPageWithPlaceholder(tester,
            bloc: homeworkPageBloc, initialTab: HomeworkTab.open);

        expect(
            find.byKey(
                ValueKey('no-homework-teacher-placeholder-for-open-homework')),
            findsOneWidget);

        homeworkPageBloc.close();
      });

New code:

      setUp(() {
        // Creating it here
        homeworkPageBloc = MockTeacherHomeworkPageBloc();
      });

      testWidgets('placeholder is shown on open homework page when no homeworks are to do',
          (tester) async {
        await pumpHomeworkPage(tester,
            bloc: homeworkPageBloc, initialTab: HomeworkTab.open);

        homeworkPageBloc.emitNewState(_noHomeworks);

        await tester.pump();

        expect(
            find.byKey(
                ValueKey('no-homework-teacher-placeholder-for-open-homework')),
            findsOneWidget);

        homeworkPageBloc.close();
      });

But when I do this the test will fail.
It fails because in the test I use homeworkPageBloc.emitNewState(_noHomeworks); which tells the bloc to emit my needed state.
When I use this method with the bloc created inside the test it works (the new state is emitted).
If I create the Bloc inside the setUp method it won't work anymore, the wanted state is only emitted after the test has already executed.
I doesn't matter how often I call pump. I can't call pumpAndSettle because of an ever-lasting animation and pumpEventQueue does not work in the widget test.

I already wasted so much time because this behavior is non-obvious. Especially if you have complexity through something like a bloc. As a normal flutter user you just don't know all this behavior. I still don't understand why this test fails exactly.

bloc.emitNewState is a method on my mock bloc which is still implemented kinda janky:

class MockTeacherHomeworkPageBloc extends HomeworkPageBloc {
  final _queuedStates = Queue<HomeworkPageState>();

  void emitNewState(HomeworkPageState state) {
    _queuedStates.add(state);
    add(LoadHomeworks());
  }

  @override
  Stream<HomeworkPageState> mapEventToState(HomeworkPageEvent event) async* {
    if (_queuedStates.isNotEmpty) {
      yield _queuedStates.removeFirst();
    }
  }

  @override
  HomeworkPageState initialState = Loading();
}

In emitNewState add(LoadHomeworks()); is only called so that mapEventToState is eventually called by the bloc framework (asynchronously - thats probably a problem?).
It's a bit of a hack but I still expected this to go smoothly especially because the call of this method has nothing to do with the bloc being set up in the setUp method or in the test body.

[maks]

@Jonas-Sander I think this is maybe catching out new Flutter developers because the way that widget tests run inside zones created by FakeAsync and then how FakeAsync works with streams is not covered at all in the widget testing intro documentation.

BUT I think you should raise that as a new issue as that's not really what this issue is about.

[Jonas-Sander]

is not covered at all in the widget testing intro documentation.

And even this assumes that everyone reads the docs.

BUT I think you should raise that as a new issue as that's not really what this issue is about.

What would the premise of the new issue be exactly? The docs? The behavior overall?

Do you have a recommendation for my case? Just skipping the setUp? Using integration_test (won't this take much longer then?) Not using Bloc? This is really confusing behavior :/
(For now I'm just gonna skip creating the bloc in setUp but I'm a bit scared for other problems with blocs and testWidgets)

[MelbourneDeveloper]

I'm not sure that the description of this problem is entirely accurate. There is no issue with awaiting Futures in Widget tests with the fake clock. Awaiting some Futures does cause hangs, but it's not all awaits. I often await Futures in widget tests because 90% of Flutter apps have Futures that need awaiting.

Here is an example of waiting for a Future. This test shows you that you can detect the before and after result of a Future completing, and it is time based with pump

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

void main() {
  testWidgets('Clicking Button Sets Flag After Wait',
      (WidgetTester tester) async {
    bool didAwait = false;

    await tester.pumpWidget(
      MaterialApp(
        home: ElevatedButton(
            onPressed: () async {
              await Future<void>.delayed(
                const Duration(seconds: 1),
              );

              didAwait = true;
            },
            child: const Text('test')),
      ),
    );

    await tester.tap(find.text('test'));
    //Proves the future hasn't completed
    expect(didAwait, false);
    await tester.pump(const Duration(seconds: 2));
    await tester.pumpAndSettle();
    //Proves that the future has completed
    expect(didAwait, true);
  });
}

And yet, this hangs

testWidgets('Await Futures', (tester) async {
  await Future.delayed(const Duration(seconds: 3));
});

I don't really know what the difference is. Both tests start Futures, but one only does so from a callback on the the widget. I feel that #139870 is related to this issue

[MelbourneDeveloper]

I just want to write a few words here to express how damaging this issue is for widget testing.

I first got this issue a few years ago. I didn't really understand how the fake clock worked. I thought it would be a good idea to simulate delays in tests because nothing in the real world is instantaneous.

I found that my tests would just hang and I would have no idea why. Later I learned ways to work around this, as you can see in the above example, but I still often tripped myself up because of this issue.

Widget testing is probably the single most important feature of the Flutter toolkit. It differentiates Flutter from other toolkits because you can test the actual UI behavior in almost no time. The fake clock makes running full tests with real user interactions feasible, unlike other toolkits that require you to use an external Black Box automated testing toolkit.

And yet, so many Flutter developers get turned off by Widget testing, and rely solely on unit testing because they find Widget Testing unapproachable and difficult. The fact that all Widgets in the Flutter repo have a vast number of widget tests prove that Widget Testing works well, but issues like this mean new Flutter devs try Widget Testing, get hangs, and get no feedback on what went wrong. This is a problem because it discourages new Flutter devs from pursuing Widget Testing.

Real attention needs to be paid to this issue.

[passsy]

Related dart-lang/test#2303

[huycozy]

Checking this issue again with the latest Flutter channels (stable/master), the test is still hanged when not using runAsync.

01:40 +0: awaiting future with value from setUp works                                                                                                                                                           

flutter doctor -v (stable and master)

[✓] Flutter (Channel stable, 3.16.7, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.16.7 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ef1af02aea (6 hours ago), 2024-01-11 15:19:26 -0600
    • Engine revision 4a585b7929
    • Dart version 3.2.4
    • DevTools version 2.28.5

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.14.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.85.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.80.0

[✓] Connected device (3 available)
    • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64  • Android 11 (API 30)
    • macOS (desktop)  • macos            • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)     • chrome           • web-javascript • Google Chrome 120.0.6099.216

[✓] Network resources
    • All expected network resources are available.

• No issues found!

[!] Flutter (Channel master, 3.19.0-6.0.pre.20, on macOS 14.1 23B74 darwin-x64, locale en-VN)
    • Flutter version 3.19.0-6.0.pre.20 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 2d93b4466f (39 minutes ago), 2024-01-16 22:42:11 -0500
    • Engine revision 1382ff79dd
    • Dart version 3.4.0 (build 3.4.0-24.0.dev)
    • DevTools version 2.31.0
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.14.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.1)
    • Android Studio at /Applications/Android Studio Hedgehog.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • android-studio-dir = /Applications/Android Studio.app/
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.85.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.80.0

[✓] Connected device (3 available)
    • iPhone (mobile) • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios            • iOS 15.8 19H370
    • macOS (desktop) • macos                                    • darwin-x64     • macOS 14.1 23B74 darwin-x64
    • Chrome (web)    • chrome                                   • web-javascript • Google Chrome 120.0.6099.234

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

[desmond206x]

Still valid for the current version 3.24.4