Include failure reason when snapshot comparisons fail #638
Conversation
…on fails to match
| let deltaFilter = CIFilter( | ||
| name: "CILabDeltaE", | ||
| parameters: [ | ||
| kCIInputImageKey: CIImage(cgImage: newCgImage), | ||
| "inputImage2": CIImage(cgImage: oldCgImage) | ||
| ] | ||
| ) | ||
| guard let deltaOutputImage = deltaFilter?.outputImage else { return false } | ||
| let extent = CGRect(x: 0, y: 0, width: oldCgImage.width, height: oldCgImage.height) | ||
| guard | ||
| let thresholdOutputImage = try? ThresholdImageProcessorKernel.apply( | ||
| withExtent: extent, | ||
| inputs: [deltaOutputImage], | ||
| arguments: [ThresholdImageProcessorKernel.inputThresholdKey: (1 - perceptualPrecision) * 100] | ||
| ) | ||
| else { return false } | ||
| let averageFilter = CIFilter( | ||
| name: "CIAreaAverage", | ||
| parameters: [ | ||
| kCIInputImageKey: thresholdOutputImage, | ||
| kCIInputExtentKey: extent | ||
| ] | ||
| ) | ||
| guard let averageOutputImage = averageFilter?.outputImage else { return false } | ||
| var averagePixel: Float = 0 | ||
| CIContext(options: [.workingColorSpace: NSNull(), .outputColorSpace: NSNull()]).render( | ||
| averageOutputImage, | ||
| toBitmap: &averagePixel, | ||
| rowBytes: MemoryLayout<Float>.size, | ||
| bounds: CGRect(x: 0, y: 0, width: 1, height: 1), | ||
| format: .Rf, | ||
| colorSpace: nil | ||
| return perceptuallyCompare( | ||
| CIImage(cgImage: oldCgImage), | ||
| CIImage(cgImage: newCgImage), | ||
| pixelPrecision: precision, | ||
| perceptualPrecision: perceptualPrecision |
There was a problem hiding this comment.
The CoreImage operations were extracted into a separate method since they were common across UIImage.swift and NSImage.swift
| format: .Rf, | ||
| colorSpace: nil | ||
| ) | ||
| let actualPerceptualPrecision = 1 - maximumDeltaE / 100 |
There was a problem hiding this comment.
@ejensen I'm not familiar with this filter's output, but when taking this for a spin in isowords I noticed that it seems to be flipped:
Should this just be maximumDeltaE / 100?
There was a problem hiding this comment.
Or maybe not, I was misremembering for this one:
Might just need to get a feel for the change and what the numbers "mean" :)
There was a problem hiding this comment.
@stephencelis Delta E should have an inverse relationship with precision. A higher Delta(Difference)E means the pixels are less similar(precise).
The wording of the error reporting might need some finessing. Seeing "Actual perceptual precision: 0.56" insinuates the entire image is significantly different. When in reality the perceptual precision being reported is the lowest of all the pixels compared, which could be a significant outlier.
There was a problem hiding this comment.
So when it says "Actual image precision 0.03083378 is less than required 0.9" in my test logs what does that mean?
There was a problem hiding this comment.
So when it says "Actual image precision 0.03083378 is less than required 0.9" in my test logs what does that mean?
That means that the lowest perceptual precision of all the pixels is 3.08%. You would need to set perceptualPrecision <= 0.0308 if you were to require all the pixels to meet the perceptual requirement (eg. setting precision: 1.0).
The failure messages could both better explain the values and provide suggestions. I'm prototyping this in the image-comparison-playground branch. It contains an iOS/macOS compatible Swift Playground where you supply two snapshots and it displays the result of the comparison along with all the intermediaries for inspection.
The output makes suggestions for values to use when images do not match:
❌ Images do not match…
The images would match if either:
1. precision is lowered to 0.9991
2. perceptualPrecision is lowered to 0.9939
3. precision is raised to 1.0 and perceptualPrecision is lowered to 0.9912
ℹ️ 99.91% of pixels match with 99.98% perceptual precision
Matching images also get suggestions when the precision values could be raised:
✅ Images match
The images would still match if:
1. precision is raised to 0.9998
2. perceptualPrecision is raised to 0.9939
3. precision is raised to 1.0 and perceptualPrecision is set to 0.9859
ℹ️ 100.00% of pixels match with 98.00% perceptual precision
There is a tradeoff between detail + precision and speed of execution. It is particularly expensive to calculate the maximum matching perceptualPrecision when precision < 1. It takes a couple of seconds to calculate with a full-screen snapshot. The other suggestions are quite fast and are likely reasonable to include in the swift-snapshot-test's failure messages.
There was a problem hiding this comment.
Actual image precision 0.03083378 is less than required 0.9
Isn't that referring to precision, not perceptualPrecision?
There was a problem hiding this comment.
So does it mean there is at least one pixel in the image that is 97% different from the same pixel on the reference image? Sorry, I still don't get what it means when it says "Actual image precision 0.03083378 is less than required 0.9" when you're looking at two identical-looking images.
Or, only 3% of the pixels in the image match the corresponding pixels in the other image enough to pass the perceptualPrecision parameter? Why is it so low when there are no perceptible differences? If it was like 80% or 90% fine but 3% I can't understand.
edit:
When in reality the perceptual precision being reported is the lowest of all the pixels compared, which could be a significant outlier.
So there's just one pixel in the image that has a precision of 3%.
There was a problem hiding this comment.
So does it mean there is at least one pixel in the image that is 97% different from the same pixel on the reference image
Yes, that is what that message means.
There was a problem hiding this comment.
I guess currently there's nothing built-in that I can use to filter out outliers?
…on fails to match (pointfreeco#638)
…on fails to match (pointfreeco#638)
…on fails to match (pointfreeco#638)
A follow-up to #628 (comment)
This PR updates the image comparison methods to report a failure message rather than just a boolean of whether the comparison passed or failed. This failure message provides more insight into the reason why a snapshot assertion failed. This is especially helpful when the image differences are subtle - such as the "ghost" differences mentioned in #628 (comment)