The app doesn't stop after failing golden test. + Error in TestGoldenComparatorProcess

[RustamG]

Steps to reproduce

1. Clone the repo. It has a sample golden test with an image that fails the test intentionally.
2. Launch iOS simulator
3. Run in the terminal flutter test integration_test/E2Es

Expected results

The test process stops. I should not kill it manually.

Actual results

The test process doesn't exit having the following output:

<< [ERROR:flutter/shell/testing/tester_main.cc(682)] Unhandled exception
<<< Exception: Golden "goldens/failing_counter.png": Pixel test failed, 0.11%, 4496px diff detected.
<<< Failure feedback can be found at /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/failures
<<< Stack trace: #0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:108:5)
<<< <asynchronous suspension>
<<< #1      main (file:///var/folders/6n/2pnrqh45205_wq6kvgs8xmq00000gn/T/flutter_tools.40hhzs/flutter_web_platform.TmZB1C/listenerHyn8yD/listener.dart:29:26)
<<< <asynchronous suspension>
<<< 

00:23 +0 -1: loading /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/ui_test.dart [E]                                                                 
  type 'Null' is not a subtype of type 'Map<String, dynamic>'
  dart:async/stream_impl.dart 1038:17                                _StreamIterator.current
  package:flutter_tools/src/test/test_golden_comparator.dart 315:27  TestGoldenComparatorProcess.getResponse
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/test_golden_comparator.dart 161:41  TestGoldenComparator._compareGoldens
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/test_golden_comparator.dart 127:28  TestGoldenComparator.compare
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/flutter_platform.dart 554:17        FlutterPlatform._listenToVmServiceForGoldens.<fn>

Code sample

Code sample

import 'package:flutter_test/flutter_test.dart';
import 'package:golden_tests/main.dart';
import 'package:integration_test/integration_test.dart';

const kGoldensFolder = 'goldens';

void main() {
  testWidgets('Counter smoke test', (WidgetTester tester) async {
    await tester.pumpWidget(const MyApp());

    expect(find.text('0'), findsOneWidget);
    expect(find.text('1'), findsNothing);

    await tester.pumpAndSettle(const Duration(seconds: 5));

    await compareScreenshot(kGoldensFolder, 'failing_counter');
  });
}

Future<void> compareScreenshot(String goldensFolder, String key) async {
  List<int> screenshotBytes =
      await IntegrationTestWidgetsFlutterBinding.ensureInitialized()
          .takeScreenshot(key);

  await expectLater(
    screenshotBytes,
    matchesGoldenFile('$goldensFolder/$key.png'),
  );
}

Screenshots or Video

Screenshots

Logs

No response

Flutter Doctor output

Doctor output

[✓] Flutter (Channel stable, 3.35.1, on macOS 15.6 24G84 darwin-arm64, locale en-RU) [334ms]
    • Flutter version 3.35.1 on channel stable at /Users/x/fvm/versions/3.35.1
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 20f8274939 (8 days ago), 2025-08-14 10:53:09 -0700
    • Engine revision 1e9a811bf8
    • Dart version 3.9.0
    • DevTools version 2.48.0
    • Feature flags: enable-web, enable-linux-desktop, enable-macos-desktop, enable-windows-desktop, enable-android, enable-ios, cli-animations, enable-lldb-debugging

[✓] Android toolchain - develop for Android devices (Android SDK version 35.0.0) [1,087ms]
    • Android SDK at /Users/x/Library/Android/sdk/
    • Emulator version 35.2.10.0 (build_id 12414864) (CL:N/A)
    • Platform android-35, build-tools 35.0.0
    • ANDROID_HOME = /Users/x/Library/Android/sdk/
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
      This is the JDK bundled with the latest Android Studio installation on this machine.
      To manually set the JDK path, use: `flutter config --jdk-dir="path/to/jdk"`.
    • Java version OpenJDK Runtime Environment (build 21.0.3+-79915917-b509.11)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 16.4) [685ms]
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16F6
    • CocoaPods version 1.16.2

[✓] Chrome - develop for the web [103ms]
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2024.2) [103ms]
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 21.0.3+-79915917-b509.11)

[✓] IntelliJ IDEA Ultimate Edition (version 2025.1.3) [102ms]
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 87.0
    • Dart plugin version 251.27623.5

[✓] IntelliJ IDEA Community Edition (version 2024.3.6) [23ms]
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.102.2) [6ms]
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (3 available) [5.7s]
    • iPad Pro 11-inch (M4) (mobile) • 19A2CC4F-464F-4482-87E2-0B29AEED90A4 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-18-6 (simulator)
    • macOS (desktop)                • macos                                • darwin-arm64   • macOS 15.6 24G84 darwin-arm64
    • Chrome (web)                   • chrome                               • web-javascript • Google Chrome 139.0.7258.139

[✓] Network resources [682ms]
    • All expected network resources are available.

• No issues found!

[darshankawar]

1. Clone the repo.

@RustamG I am unable to access the repo. Maybe it is private.

The test process doesn't exit having the following output:

<< [ERROR:flutter/shell/testing/tester_main.cc(682)] Unhandled exception
<<< Exception: Golden "goldens/failing_counter.png": Pixel test failed, 0.11%, 4496px diff detected.
<<< Failure feedback can be found at /Users/x/Documents/Flutter/golden_tests/integration_test/E2Es/failures
<<< Stack trace: #0      LocalFileComparator.compare (package:flutter_test/src/_goldens_io.dart:108:5)
<<< <asynchronous suspension>
<<< #1      main (file:///var/folders/6n/2pnrqh45205_wq6kvgs8xmq00000gn/T/flutter_tools.40hhzs/flutter_web_platform.TmZB1C/listenerHyn8yD/listener.dart:29:26)
<<< <asynchronous suspension>

Based on this error log, please check if your case is same as #92590 or not.

[RustamG]

@darshankawar My bad. Just made it public.

As for #92590 - it is different. In my case I made the golden image different intentionally to demonstrate another issue - that the app and test don't stop after failure.

[darshankawar]

Thanks for the update. I was able to replicate the reported behavior on latest sdk versions getting same error.

stable : 3.35.1
master : 3.36.0-1.0.pre-211