Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent PNG encoding by Image.toByteData() causes false positives in golden file tests #30036

Closed
apaatsio opened this issue Mar 27, 2019 · 3 comments
Labels

Comments

@apaatsio
Copy link
Contributor

@apaatsio apaatsio commented Mar 27, 2019

Problem

I have a situation where matchesGoldenFile tests fail when golden files are created on one machine and the tests are run on another. Both are Linux machines with the same Flutter and Dart versions.

Cause

I tracked it down to this line:

final ByteData bytes = await image.toByteData(format: ui.ImageByteFormat.png)

This seems to produce a PNG presentation that differs bytewise on different machines even though the images are identical pixelwise.

The matchesGoldenFile tests compare the PNG images bytewise and this causes the test to fail even with pixelwise identical files.

Future<bool> compare(Uint8List imageBytes, Uri golden) async {
final File goldenFile = _getFile(golden);
if (!goldenFile.existsSync()) {
throw test_package.TestFailure('Could not be compared against non-existent file: "$golden"');
}
final List<int> goldenBytes = await goldenFile.readAsBytes();
return _areListsEqual<int>(imageBytes, goldenBytes);
}

To reproduce

I created this short code that creates an image and prints its raw hash and png hash.

import 'dart:typed_data';
import 'dart:ui';
import 'package:collection/collection.dart';

void main() async {
  final pictureRecorder = PictureRecorder();
  Canvas(pictureRecorder).drawCircle(
    const Offset(100, 100),
    100,
    Paint()..color = const Color.fromARGB(255, 255, 0, 0),
  );
  final picture = pictureRecorder.endRecording();
  final image = await picture.toImage(200, 200);

  final hashFunc = const ListEquality().hash;
  final pngByteData = await image.toByteData(format: ImageByteFormat.png);
  final pngHash = hashFunc(Uint8List.view(pngByteData.buffer));
  final rawByteData = await image.toByteData(format: ImageByteFormat.rawRgba);
  final rawHash = hashFunc(Uint8List.view(rawByteData.buffer));
  print('png hash: $pngHash');
  print('raw hash: $rawHash');
}

Test runs

Machine 1:

❯ flutter test image_test.dart 
00:01 +0: loading /home/antti/projects/correct_pitch/image_test.dart                                                                                                                                     
png hash: 1951256462
raw hash: 1641621570
No tests ran.

Machine 2:

❯ flutter test image_test.dart 
00:01 +0: loading /home/antti/projects/correct_pitch/image_test.dart                                                                                                                                     
png hash: 1038304024
raw hash: 1641621570
No tests ran.

As you can see, the raw hashes are identical, i.e. the images are identical pixelwise. But the PNG hashes differ, i.e. the bytes in the PNG presentation are not consistent.

flutter doctor -v

Machine 1:

[✓] Flutter (Channel beta, v1.3.8, on Linux, locale en_US.UTF-8)
    • Flutter version 1.3.8 at /home/antti/tools/flutter
    • Framework revision e5b1ed7a7f (3 weeks ago), 2019-03-06 14:23:37 -0800
    • Engine revision f4951df193
    • Dart version 2.2.1 (build 2.2.1-dev.0.0 571ea80e11)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /opt/android-sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /opt/android-sdk
    • Java binary at: /opt/android-studio/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[✓] Android Studio (version 3.3)
    • Android Studio at /opt/android-studio
    • Flutter plugin version 33.3.1
    • Dart plugin version 182.5215
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[!] Connected device
    ! No devices available

! Doctor found issues in 1 category.

Machine 2:

[✓] Flutter (Channel beta, v1.3.8, on Linux, locale en_US.UTF-8)
    • Flutter version 1.3.8 at /home/antti/tools/flutter
    • Framework revision e5b1ed7a7f (3 weeks ago), 2019-03-06 14:23:37 -0800
    • Engine revision f4951df193
    • Dart version 2.2.1 (build 2.2.1-dev.0.0 571ea80e11)

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /home/antti/Android/Sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /home/antti/Android/Sdk
    • Java binary at: /usr/bin/java
    • Java version OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4)
    • All Android licenses accepted.

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.io/setup/#android-setup for detailed instructions).

[✓] VS Code (version 1.32.3)
    • VS Code at /usr/share/code
    • Flutter extension version 2.24.0

[!] Connected device
    ! No devices available

! Doctor found issues in 2 categories.
@apaatsio

This comment has been minimized.

Copy link
Contributor Author

@apaatsio apaatsio commented Mar 27, 2019

Here's a real-life example of golden files that are produced by two different computers. The pixels are identical but bytes are not.

Both images are created with flutter test --update-goldens on the exact same code base with the exact same Flutter and Dart versions as described in the original bug report.

Note how the md5sums differ but the images are identical according to the ImageMagick signature.

golden file machine1 machine2
machine machine 1 machine 2
md5sum 810108768907135108c0ba33c0271b3f 017805ffb21c33e85845412900913a89
ImageMagick signature (identify -format "%#") 68f86f32fccacc0f9a270a0a90d2ff26 ce0af5ad1b59f1c7ae39d3e93b10e052 68f86f32fccacc0f9a270a0a90d2ff26 ce0af5ad1b59f1c7ae39d3e93b10e052
@tp

This comment has been minimized.

Copy link

@tp tp commented Jun 19, 2019

We also run into this constantly, where images change but are visually the same.

Though personally I see it also happening on one machine, where the same test produces different outputs, even though I am unable to spot any difference with tooling.

(I don't have a small repro, but could send the files if anyone wants to take a look at the output.)

Edit: Using magick identify -format "%#" filename I do get 2 different hashes, even though visually no tool would show a single pixel of difference:

bdc3540f6f5ede578979101d7091ecb07003d769a732642b0e13d4f427cb7a47
0a4b730d76d35d51d06226a94122f218a517a47e05e972faee3e8ae27a48d3e

@apaatsio

This comment has been minimized.

Copy link
Contributor Author

@apaatsio apaatsio commented Oct 7, 2019

Closing this issue because by all logic this has been fixed by #38473. Although I can't verify because I don't have access to the original environment anymore to reproduce the issue.

@apaatsio apaatsio closed this Oct 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.