Unit tests don't fail if an exception is thrown inside ChangeNotifier callback

[flbaue]

Steps to Reproduce

1. Create a unit test (not widget test) to test a change notifier implementation.
2. Add a listener with expectAsync0 that will throw an exception when called

Actual results:

• The exception is logged to the console, but the test completes successful.

Expected results:

• The test should fail

I assume, that the issue is, that inside ChangeNotifier callback errors are caught and not propagated to the test runner.
If I set the error reporter explicitly like so:

 FlutterError.onError = (details) => reportTestException(
            details,
            'should set error when summaries are empty',
          );

At the beginning of the test, the test fails as I would expect it to.

Code sample

// ChangeNotifier
class MyController extends ChangeNotifier {
  bool isLoading = false;
Future<void> fetch() async {
    isLoading = true;
    notifyListeners();
   await Future.delayed(Duration(seconds:1, () {});
    isLoading = false;
    notifyListeners();
  }
}

// test
void main() {
  test('my test', () async {
   final controller = MyController();
   controller.addListener(expectAsync0((){
     expect(1,2); // supposed to fail for demonstration
    }));
   await controller.fetch();
  })
}

Logs

This will log something like this:

═╡ EXCEPTION CAUGHT BY FOUNDATION LIBRARY ╞════════════════════════════════════════════════════════
The following TestFailure was thrown while dispatching notifications for MyController:
Expected: <2>
  Actual: <1>

...

✓ MyController my test

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.3.9, on macOS 13.0.1 22A400 darwin-x64, locale de-DE)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.3)
[✓] IntelliJ IDEA Community Edition (version 2021.1.1)
[✓] VS Code (version 1.73.1)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

Update 1:
if I use testWidgets(...) instead of test(...) the test behaves as expected and fails. So it seems to me that the configuration of the error reporter is missing inside the test method.

Maybe this is somehow intended, but at least from my perspective it is not clear, why the test of a plain ChangeNotifier should be a widget test.

[exaby73]

Triage report

I can reproduce this issue on Master (3.7.0-4.0.pre.38). The error is logged to the console, but all tests pass.

Code Sample

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';

class MyController extends ChangeNotifier {
  bool isLoading = false;
  Future<void> fetch() async {
    isLoading = true;
    notifyListeners();
    await Future.delayed(const Duration(seconds: 1), () {});
    isLoading = false;
    notifyListeners();
  }
}

// test
void main() {
  test('my test', () async {
    final controller = MyController();
    controller.addListener(expectAsync0(() {
      expect(1, 2); // supposed to fail for demonstration
    }));
    await controller.fetch();
  });
}

Logs

══╡ EXCEPTION CAUGHT BY FOUNDATION LIBRARY ╞════════════════════════════════════════════════════════
The following TestFailure was thrown while dispatching notifications for MyController:
Expected: <2>
  Actual: <1>

When the exception was thrown, this was the stack:
#0      fail (package:test_api/src/expect/expect.dart:134:31)
#1      _expect (package:test_api/src/expect/expect.dart:129:3)
#2      expect (package:test_api/src/expect/expect.dart:46:3)
#3      expect (package:flutter_test/src/widget_tester.dart:460:16)
#4      main.<anonymous closure>.<anonymous closure> (file:///Users/nabeelparkar/dev/nevercode/issue_116653/test/widget_test.dart:21:7)
#5      Function._apply (dart:core-patch/function_patch.dart:11:73)
#6      Function.apply (dart:core-patch/function_patch.dart:34:12)
#7      _ExpectedFunction._run (package:test_api/src/expect/expect_async.dart:195:23)
#8      _ExpectedFunction.max6 (package:test_api/src/expect/expect_async.dart:178:7)
#9      _ExpectedFunction.max0 (package:test_api/src/expect/expect_async.dart:144:15)
#10     ChangeNotifier.notifyListeners (package:flutter/src/foundation/change_notifier.dart:381:24)
#11     MyController.fetch (file:///Users/nabeelparkar/dev/nevercode/issue_116653/test/widget_test.dart:9:5)
#12     main.<anonymous closure> (file:///Users/nabeelparkar/dev/nevercode/issue_116653/test/widget_test.dart:23:22)
#13     Declarer.test.<anonymous closure>.<anonymous closure> (package:test_api/src/backend/declarer.dart:215:19)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

The MyController sending notification was:
  Instance of 'MyController'
════════════════════════════════════════════════════════════════════════════════════════════════════
Another exception was thrown: Callback called more times than expected (1).
00:05 +1: my test                                                                                                                        
00:05 +1: All tests passed!