Merge pull request #3 from ChrisRega/0.3.0-dev

0.3.0 dev

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "image-compare"
-version = "0.2.5"
+version = "0.3.0"
 edition = "2021"
 authors = ["Christopher Regali <christopher.regali@vdop.org>"]
 license = "MIT"

README.md

@@ -5,33 +5,53 @@
 [![Coverage Status](https://coveralls.io/repos/github/ChrisRega/image-compare/badge.svg?branch=main)](https://coveralls.io/github/ChrisRega/image-compare?branch=main)
 [![License](https://img.shields.io/badge/license-MIT-blue?style=flat)](LICENSE)
 
-Simple image comparison in rust based on the image crate
+Image comparison in rust based on the image crate
 
-Note that this crate is heavily work in progress. Algorithms are neither cross-checked not particularly fast yet.
-Everything is implemented in plain CPU with rayon multithreading. 
-SIMD is under investigation on a feature branch (simd-experimental).
+- Note that this crate is still work in progress. 
+- Algorithms are not cross-checked.
+- Everything is implemented in plain CPU with rayon multithreading and seems to perform just fine on modern processors.
+Neither [memory optimizations](https://actix.vdop.org/view_post?post_num=10) nor [SIMD](https://actix.vdop.org/view_post?post_num=8) seemed to provide any remarkable improvement.
 
-### Supported now:
-- Comparing grayscale and rgb images by structure
-  - By RMS - score is calculated by: <img src="https://render.githubusercontent.com/render/math?math=1-\sqrt{\frac{(\sum_{x,y=0}^{x,y=w,h}\left(f(x,y)-g(x,y)\right)^2)}{w*h}}"> 
+## Comparing grayscale images
+### By structure
+  - By RMS - score is calculated by: $1-\sqrt{\frac{\sum_{x,y=0}^{x,y=w,h}\left(f(x,y)-g(x,y)\right)^2}{w*h}}$
   - By MSSIM
-    - SSIM is implemented as described on [wikipedia](https://en.wikipedia.org/wiki/Structural_similarity): <img src="https://render.githubusercontent.com/render/math?math=\mathrm{SSIM}(x,y)={\frac {(2\mu _{x}\mu _{y}+c_{1})(2\sigma _{xy}+c_{2})}{(\mu _{x}^{2}+\mu _{y}^{2}+c_{1})(\sigma _{x}^{2}+\sigma _{y}^{2}+c_{2})}}"> 
+    - SSIM is implemented as described on [wikipedia](https://en.wikipedia.org/wiki/Structural_similarity): $\mathrm{SSIM}(x,y)={\frac{(2\mu_{x}\mu_{y}+c_{1})(2\sigma_{xy}+c_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+c_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+c_{2})}}$ 
     - MSSIM is calculated by using 8x8 pixel windows for SSIM and averaging over the results
-  - RGB type images are split to R,G and B channels and processed separately. 
-    - The worst of the color results is propagated as score but a float-typed RGB image provides access to all values.
-    - As you can see in the gherkin tests this result is not worth it currently, as it takes a lot more time
-    - It could be improved, by not just propagating the individual color-score results but using the worst for each pixel
-    - This approach is implemented in hybrid-mode, see below
-  - "hybrid comparison"
-    - Splitting the image to YUV colorspace according to T.871
-    - Processing the Y channel with MSSIM
-    - Comparing U and V channels via RMS
-    - Recombining the differences to a nice visualization image
-    - Score is calculated as: <img src="https://render.githubusercontent.com/render/math?math=\mathrm{score}=\mathrm{avg}_{x,y}\left(\mathrm{min}\left[\Delta \mathrm{MSSIM}(x,y),1- \sqrt{(1-\Delta RMS(u,x,y))^2 + (1-\Delta RMS(v,x,y))^2}\right]\right)">
-    - This allows for a good separation of color differences and structure differences
-- Comparing grayscale images by histogram
+### By histogram
   - Several distance metrics implemented see [OpenCV docs](https://docs.opencv.org/4.5.5/d8/dc8/tutorial_histogram_comparison.html):
-    - Correlation <img src="https://render.githubusercontent.com/render/math?math=d(H_1,H_2) = \frac{\sum_I (H_1(I) - \bar{H_1}) (H_2(I) - \bar{H_2})}{\sqrt{\sum_I(H_1(I) - \bar{H_1})^2 \sum_I(H_2(I) - \bar{H_2})^2}}">
-    - Chi-Square <img src="https://render.githubusercontent.com/render/math?math=d(H_1,H_2) = \sum _I \frac{\left(H_1(I)-H_2(I)\right)^2}{H_1(I)}">
-    - Intersection <img src="https://render.githubusercontent.com/render/math?math=d(H_1,H_2) = \sum _I \min (H_1(I), H_2(I))">
-    - Hellinger distance <img src="https://render.githubusercontent.com/render/math?math=d(H_1,H_2) = \sqrt{1 - \frac{1}{\sqrt{\int{H_1} \int{H_2}}} \sum_I \sqrt{H_1(I) \cdot H_2(I)}}">
+  - Correlation $d(H_1,H_2) = \frac{\sum_I (H_1(I) - \bar{H_1}) (H_2(I) - \bar{H_2})}{\sqrt{\sum_I(H_1(I) - \bar{H_1})^2 \sum_I(H_2(I) - \bar{H_2})^2}}$
+  - Chi-Square $d(H_1,H_2) = \sum _I \frac{\left(H_1(I)-H_2(I)\right)^2}{H_1(I)}$
+  - Intersection $d(H_1,H_2) = \sum _I \min (H_1(I), H_2(I))$
+  - Hellinger distance $d(H_1,H_2) = \sqrt{1 - \frac{1}{\sqrt{\int{H_1} \int{H_2}}} \sum_I \sqrt{H_1(I) \cdot H_2(I)}}$
+
+## Comparing RGB(A)
+### By structure: RMS, SSIM
+  - RGB type images are split to R,G and B channels and processed separately.
+  - The worst of the color results is propagated as score but a float-typed RGB image provides access to all values.
+  - As you can see in the gherkin tests this result is not worth it currently, as it takes a lot more time
+  - It could be improved, by not just propagating the individual color-score results but using the worst for each pixel
+  - This approach is implemented in hybrid-mode, see below
+### By structure: "Hybrid Comparison"
+  - Splitting the image to YUV colorspace according to T.871
+  - Processing the Y channel with MSSIM
+  - Comparing U and V channels via RMS
+  - Recombining the differences to a nice visualization image
+  - RGB Score is calculated as: $\mathrm{score}=\mathrm{avg}_{x,y}\left(\mathrm{min}\left[\Delta \mathrm{MSSIM}(Y,x,y),\sqrt{(\Delta RMS(U,x,y))^2 + (\Delta RMS(V,x,y))^2}\right]\right)$
+  - RGBA can either be premultiplied with a specifiable background color using `rgba_blended_hybrid_compare`
+  - Otherwise, for `rgba_hybrid_compare` the $\alpha$ channel is also compared using MSSIM and taken into account.
+  - The average alpha of each pixel $\bar{\alpha}(x,y) = 1/2 (\alpha_1(x,y) + \alpha_2(x,y))$ is then used as a linear weighting factor
+  - RGBA Score is calculated as: $\mathrm{score}=\mathrm{avg}_{x,y}\left(1/\bar{\alpha} \cdot \mathrm{min}\left[\Delta \mathrm{MSSIM}(Y,x,y),\sqrt{(\Delta RMS(U,x,y))^2 + (\Delta RMS(V,x,y))^2}, \Delta \mathrm{RMS}(\alpha,x,y)\right]\right)$
+  - Edge cases RGBA: $\mathrm{score} \in (0, 1)$ and $\mathrm{score} = 1.0$ if $\bar{\alpha} = 0.0$
+  - This allows for a good separation of color differences and structure differences for both RGB and RGBA
+  - Interpretation of the diff-images:
+    - RGB: Red contains structure differences, Green and Blue the color differences, the more color, the higher the diff
+    - RGBA: Same as RGB but alpha contains the inverse of the alpha-diffs. If something is heavily translucent, the alpha was so different, that differentiating between color and structure difference would be difficult. Also, minimum alpha is clamped at 0.1, so you can still see all changes.
+
+Changelog:
+0.3.0:
+- An error was found in hybrid RGB compare in 0.2.x that over-weighted color differences. Numbers in tests were adjusted
+  - Influence was very small for most images but noticeable for the color-filtered one which yields much higher similarity now
+- Added two methods for RGBA comparison
+- Added GitHub inline latex for equations instead of embedded images - fixes dark theme rendering
+- Made API more intuitive

src/colorization.rs

@@ -0,0 +1,95 @@
+use image::{DynamicImage, GrayImage, ImageBuffer, Luma, Rgb, RgbImage, Rgba, RgbaImage};
+
+/// a single-channel f32 typed image containing a result-score for each pixel
+pub type GraySimilarityImage = ImageBuffer<Luma<f32>, Vec<f32>>;
+/// a three-channel f32 typed image containing a result-score per color channel for each pixel
+pub type RGBSimilarityImage = ImageBuffer<Rgb<f32>, Vec<f32>>;
+/// a four-channel f32 typed image containing a result-score per color channel for each pixel
+pub type RGBASimilarityImage = ImageBuffer<Rgba<f32>, Vec<f32>>;
+
+#[derive(Debug)]
+pub enum SimilarityImage {
+    Gray(GraySimilarityImage),
+    RGB(RGBSimilarityImage),
+    RGBA(RGBASimilarityImage),
+}
+
+impl From<GraySimilarityImage> for SimilarityImage {
+    fn from(value: GraySimilarityImage) -> Self {
+        SimilarityImage::Gray(value)
+    }
+}
+impl From<RGBASimilarityImage> for SimilarityImage {
+    fn from(value: RGBASimilarityImage) -> Self {
+        SimilarityImage::RGBA(value)
+    }
+}
+impl From<RGBSimilarityImage> for SimilarityImage {
+    fn from(value: RGBSimilarityImage) -> Self {
+        SimilarityImage::RGB(value)
+    }
+}
+
+fn gray_map(img: &GraySimilarityImage) -> DynamicImage {
+    let mut img_gray = GrayImage::new(img.width(), img.height());
+    for row in 0..img.height() {
+        for col in 0..img.width() {
+            let new_val = img.get_pixel(col, row)[0].clamp(0., 1.) * 255.;
+            img_gray.put_pixel(col, row, Luma([new_val as u8]));
+        }
+    }
+    img_gray.into()
+}
+
+fn to_color_map(img: &RGBSimilarityImage) -> DynamicImage {
+    let mut img_rgb = RgbImage::new(img.width(), img.height());
+    for row in 0..img.height() {
+        for col in 0..img.width() {
+            let pixel = img.get_pixel(col, row);
+            let mut new_pixel = [0u8; 3];
+            for channel in 0..3 {
+                new_pixel[channel] = (pixel[channel].clamp(0., 1.) * 255.) as u8;
+            }
+            img_rgb.put_pixel(col, row, Rgb(new_pixel));
+        }
+    }
+    img_rgb.into()
+}
+
+fn to_color_map_rgba(img: &RGBASimilarityImage) -> DynamicImage {
+    let mut img_rgba = RgbaImage::new(img.width(), img.height());
+    for row in 0..img.height() {
+        for col in 0..img.width() {
+            let pixel = img.get_pixel(col, row);
+            let mut new_pixel = [0u8; 4];
+            for channel in 0..4 {
+                new_pixel[channel] = (pixel[channel].clamp(0., 1.) * 255.) as u8;
+            }
+            img_rgba.put_pixel(col, row, Rgba(new_pixel));
+        }
+    }
+    img_rgba.into()
+}
+
+impl SimilarityImage {
+    pub fn to_color_map(&self) -> DynamicImage {
+        match self {
+            SimilarityImage::Gray(gray) => gray_map(gray),
+            SimilarityImage::RGB(rgb) => to_color_map(rgb),
+            SimilarityImage::RGBA(rgba) => to_color_map_rgba(rgba),
+        }
+    }
+}
+
+#[derive(Debug)]
+/// the resulting struct containing both an image of per pixel diffs as well as an average score
+pub struct Similarity {
+    /// Contains the resulting differences per pixel if applicable
+    /// The buffer will contain the resulting values of the respective algorithms:
+    /// - RMS will be between 0. for all-white vs all-black and 1.0 for identical
+    /// - SSIM usually is near 1. for similar, near 0. for different but can take on negative values for negative covariances
+    /// - Hybrid mode will be inverse: 0. means no difference, 1.0 is maximum difference. For details see [`crate::hybrid::rgb_hybrid_compare`]    
+    pub image: SimilarityImage,
+    /// the average score of the image
+    pub score: f64,
+}

src/hybrid.rs

@@ -1,8 +1,70 @@
+use crate::colorization::{GraySimilarityImage, RGBASimilarityImage, RGBSimilarityImage};
 use crate::prelude::*;
-use crate::{gray_similarity_structure, Decompose};
+use crate::squared_error::root_mean_squared_error_simple;
+use crate::ssim::ssim_simple;
+use crate::utils::{blend_alpha, split_rgba_to_yuva};
+use crate::Decompose;
+use image::{Rgba, RgbaImage};
 use itertools::izip;
+use std::borrow::Cow;
 
-fn merge_similarity_channels_yuv(input: &[GraySimilarityImage; 3]) -> RGBSimilarity {
+fn merge_similarity_channels_yuva(
+    input: &[GraySimilarityImage; 4],
+    alpha: &GrayImage,
+    alpha_second: &GrayImage,
+) -> Similarity {
+    const ALPHA_VIS_MIN: f32 = 0.1;
+
+    let mut image = RGBASimilarityImage::new(input[0].width(), input[0].height());
+    let mut deviation = Vec::new();
+    deviation.resize((input[0].width() * input[0].height()) as usize, 0.0);
+    izip!(
+        image.pixels_mut(),
+        input[0].pixels(),
+        input[1].pixels(),
+        input[2].pixels(),
+        input[3].pixels(),
+        alpha.pixels(),
+        alpha_second.pixels(),
+        deviation.iter_mut()
+    )
+    .for_each(
+        |(rgba, y, u, v, a_d, alpha_source, alpha_source_second, deviation)| {
+            let y = y[0].clamp(0.0, 1.0);
+            let u = u[0].clamp(0.0, 1.0);
+            let v = v[0].clamp(0.0, 1.0);
+            let a_d = a_d[0].clamp(0.0, 1.0);
+            let alpha_bar = (alpha_source[0] as f32 + alpha_source_second[0] as f32) / (2. * 255.);
+            let alpha_bar = if alpha_bar.is_finite() {
+                alpha_bar
+            } else {
+                1.0
+            };
+
+            let color_diff = ((u).powi(2) + (v).powi(2)).sqrt().clamp(0.0, 1.0);
+            let min_sim = y.min(color_diff).min(a_d);
+            //the lower the alpha the less differences are visible in color and structure (and alpha)
+
+            let dev = if alpha_bar > 0. {
+                (min_sim / alpha_bar).clamp(0., 1.)
+            } else {
+                1.0
+            };
+            let alpha_vis = (ALPHA_VIS_MIN + a_d * (1.0 - ALPHA_VIS_MIN)).clamp(0., 1.);
+
+            *deviation = dev;
+            *rgba = Rgba([1. - y, 1. - u, 1. - v, alpha_vis]);
+        },
+    );
+
+    let score = deviation.iter().sum::<f32>() as f64 / deviation.len() as f64;
+    Similarity {
+        image: image.into(),
+        score,
+    }
+}
+
+fn merge_similarity_channels_yuv(input: &[GraySimilarityImage; 3]) -> Similarity {
     let mut image = RGBSimilarityImage::new(input[0].width(), input[0].height());
     let mut deviation = Vec::new();
     deviation.resize((input[0].width() * input[0].height()) as usize, 0.0);
@@ -17,14 +79,87 @@ fn merge_similarity_channels_yuv(input: &[GraySimilarityImage; 3]) -> RGBSimilar
         let y = y[0].clamp(0.0, 1.0);
         let u = u[0].clamp(0.0, 1.0);
         let v = v[0].clamp(0.0, 1.0);
-        let color_diff = 1. - (1. - u.powi(2) + 1. - v.powi(2)).sqrt().clamp(0.0, 1.0);
+        let color_diff = ((u).powi(2) + (v).powi(2)).sqrt().clamp(0.0, 1.0);
         //f32 for keeping numerical stability for hybrid compare in 0.2.-branch
         *deviation += y.min(color_diff);
         *rgb = Rgb([1. - y, 1. - u, 1. - v]);
     });
 
     let score = deviation.iter().sum::<f32>() as f64 / deviation.len() as f64;
-    RGBSimilarity { image, score }
+    Similarity {
+        image: image.into(),
+        score,
+    }
+}
+
+/// Hybrid comparison for RGBA images.
+/// Will do MSSIM on luma, then RMS on U and V and alpha channels.
+/// The calculation of the score is then pixel-wise the minimum of each pixels similarity.
+/// To account for perceived indifference in lower alpha regions, this down-weights the difference
+/// linearly with mean alpha channel.
+pub fn rgba_hybrid_compare(
+    first: &RgbaImage,
+    second: &RgbaImage,
+) -> Result<Similarity, CompareError> {
+    if first.dimensions() != second.dimensions() {
+        return Err(CompareError::DimensionsDiffer);
+    }
+
+    let first = split_rgba_to_yuva(first);
+    let second = split_rgba_to_yuva(second);
+
+    let (_, mssim_result) = ssim_simple(&first[0], &second[0])?;
+    let (_, u_result) = root_mean_squared_error_simple(&first[1], &second[1])?;
+    let (_, v_result) = root_mean_squared_error_simple(&first[2], &second[2])?;
+
+    let (_, alpha_result) = root_mean_squared_error_simple(&first[3], &second[3])?;
+
+    let results = [mssim_result, u_result, v_result, alpha_result];
+
+    Ok(merge_similarity_channels_yuva(
+        &results, &first[3], &second[3],
+    ))
+}
+
+/// A wrapper class accepting both RgbaImage and RgbImage for the blended hybrid comparison
+pub enum BlendInput<'a> {
+    /// This variant means that the image is already alpha pre-blended and therefore RGB
+    PreBlended(&'a RgbImage),
+    /// This variant means that the image still needs to be blended with a certain background
+    RGBA(&'a RgbaImage),
+}
+
+impl<'a> BlendInput<'a> {
+    fn into_blended(self, background: Rgb<u8>) -> Cow<'a, RgbImage> {
+        match self {
+            BlendInput::PreBlended(image) => Cow::Borrowed(image),
+            BlendInput::RGBA(rgba) => Cow::Owned(blend_alpha(rgba, background)),
+        }
+    }
+}
+
+impl<'a> From<&'a RgbImage> for BlendInput<'a> {
+    fn from(value: &'a RgbImage) -> Self {
+        BlendInput::PreBlended(value)
+    }
+}
+
+impl<'a> From<&'a RgbaImage> for BlendInput<'a> {
+    fn from(value: &'a RgbaImage) -> Self {
+        BlendInput::RGBA(value)
+    }
+}
+
+/// This processes the RGBA images be pre-blending the colors with the desired background color.
+/// It's faster then the full RGBA similarity and more intuitive.
+pub fn rgba_blended_hybrid_compare(
+    first: BlendInput,
+    second: BlendInput,
+    background: Rgb<u8>,
+) -> Result<Similarity, CompareError> {
+    let first = first.into_blended(background);
+    let second = second.into_blended(background);
+    rgb_hybrid_compare(&first, &second)
 }
 
 /// Comparing structure via MSSIM on Y channel, comparing color-diff-vectors on U and V summing the squares
@@ -35,33 +170,18 @@ fn merge_similarity_channels_yuv(input: &[GraySimilarityImage; 3]) -> RGBSimilar
 /// This leads to a nice visualization of color and structure differences - with structural differences (meaning gray mssim diffs) leading to red rectangles
 /// and and the u and v color diffs leading to color-deviations in green, blue and cyan
 /// All-black meaning no differences
-pub fn rgb_hybrid_compare(
-    first: &RgbImage,
-    second: &RgbImage,
-) -> Result<RGBSimilarity, CompareError> {
+pub fn rgb_hybrid_compare(first: &RgbImage, second: &RgbImage) -> Result<Similarity, CompareError> {
     if first.dimensions() != second.dimensions() {
         return Err(CompareError::DimensionsDiffer);
     }
 
     let first_channels = first.split_to_yuv();
     let second_channels = second.split_to_yuv();
-    let mssim_result = gray_similarity_structure(
-        &Algorithm::MSSIMSimple,
-        &first_channels[0],
-        &second_channels[0],
-    )?;
-    let u_result = gray_similarity_structure(
-        &Algorithm::RootMeanSquared,
-        &first_channels[1],
-        &second_channels[1],
-    )?;
-    let v_result = gray_similarity_structure(
-        &Algorithm::RootMeanSquared,
-        &first_channels[2],
-        &second_channels[2],
-    )?;
-
-    let results = [mssim_result.image, u_result.image, v_result.image];
+    let (_, mssim_result) = ssim_simple(&first_channels[0], &second_channels[0])?;
+    let (_, u_result) = root_mean_squared_error_simple(&first_channels[1], &second_channels[1])?;
+    let (_, v_result) = root_mean_squared_error_simple(&first_channels[2], &second_channels[2])?;
+
+    let results = [mssim_result, u_result, v_result];
 
     Ok(merge_similarity_channels_yuv(&results))
 }