False negative comparison results

[DiegoSanchezE]

Describe the bug
I'm getting some false-negative, image comparison results, even though the diff images don't show much difference or any difference at all.

Potentially related to the how the report is generated after compare images is used or the number of pictures being compared.
I suspect the alphabetical sorting of the images names might be playing some role here.
That is, if you have the list below:

img1
img2
img3
img10
img11
img12

and you sort it alphabetically, you get this:

img1
img10
img11
img12
img2
img3

...just a thought.

To Reproduce
I'll describe exactly my scenario, but there might be a general way to repro this.

1. Crate a TC that captures 16 different baseline images. Doesn't matter if full screen or element.
2. Run the TC in test mode.
3. View the results.

Expected behavior

• The test will fail, returning the Image dissimilarity exceeds threshold error.
• The visualReport.html will in alphabetical order. That is the results are order like so:

img1
img10
img11
img12
img13
img14
img15
img16
img2
img3
...
img9

• The failed results will belong to img10 and img9, always.
• The diff images for img10 and img9 would be mostly or completely white (tiny or no difference).

Screenshots and files:

• Comparison result for img10:
  img10.png.txt
• Actual img10:
  
• Baseline img10
  
• Diff img10, the difference is just dust which wouldn't amount to the magnitude of the reported dissimilarity:
  

Same case for img9. Dissimilarity: 0.15<3.74. Diff image:

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Chrome
• Version 70.0.3538.110 (Official Build) (64-bit)

Additional context
I will reduce the image count to 10 and later to 9, check the results and report back if I find any more clues.
Thanks!

[DiegoSanchezE]

Nevermind, this is not a library issue. The "issue" is consistent on those pictures above only.
If those are NOT in the test, the test passes, no matter in what order the pictures are or how many there are.
Closing this issue request.

[jessezach]

This seems like something to do with imagemagick? Let me test these images manually with imagemagick.

[DiegoSanchezE]

Don't worry about it, Jesse. It looks like the interaction of an one of my layers with Mapbox's layers causes some random noise. Can't really explain it yet, but it's definitely not an issue with the library. Thanks, tho On Dec 4, 2018 20:17, "Jesse Zacharias" <notifications@github.com> wrote: This seems like something to do with imagemagick? Let me test these images manually with imagemagick. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATe4_tbPza_c8nkIBbFS0JoonJ9QGDrdks5u10jJgaJpZM4ZBMNB> .