SSIM result is different from pytorch-msssim.ssim

[Pixie8888]

Hi,
I use both pytorch_mssim.ssim and skimage.measure.compare_ssim to compute ssim, but the results are different. For example, ssim evaluation on an image sequence:
pytorch_msssim.ssim: [0.9655, 0.9500, 0.9324, 0.9229, 0.9191, 0.9154]
skimage.measure.compare_ssim: [0.97794482, 0.96226299, 0.948432, 0.9386946, 0.93113704, 0.92531453]

Why will this happen?

[grlee77]

I think the default settings in scikit-image likely do not match those in pytorch_mssim. Can you confirm if you are following the suggestion in the docstring notes:

To match the implementation of Wang et. al. [1], set gaussian_weights to True, sigma to 1.5, and use_sample_covariance to False

This type of thing has come up in the past and we have done a decent amount of validation vs. 3rd party implementations (MATLAB and a C-based implementation published as part of an IPOL paper):
(examples: #5192, #5007, #4985, #4743, #4278, #3942)

[Pixie8888]

Hi, I use the evaluation code below:
for skimage.measure.compare_ssim, I calculate ssim along each channel, then average them. version=0.17.2

gt[t][i].shape = (b,3,h,w), value range(0,1) numpy float16
result:(0.9767,0.9038,0.8849)

for pytorch_msssim: from pytorch_msssim import ssim as ssim_func

img[opt.n_past:][t].shape = (b,3,h,w), value range(0,1) torch.cuda.FloatTensor
result:(0.9688, 0.8888, 0.8743)
Why are results still different?

[grlee77]

After looking briefly at pytorch-msssim, I understand the difference now. The issue is that these are not the same algorithm and should not be expected to match!

pytorch-msssim implements a variant of SSIM known as "multi-scale" SSIM (MS-SSIM) while the one here is the original SSIM method. (the variable mssim in our code is used to refer to the "mean" SSIM over the image which is the value people usually report in publications. the "m" there does not indicate multi-scale)

Thus the implementation here should be compared to either PyTorch Ignite's SSIM:
https://pytorch.org/ignite/generated/ignite.metrics.SSIM.html
or the SSIM implementation here:
https://github.com/Po-Hsun-Su/pytorch-ssim
and not to MS-SSIM implementations.

I will close this issue as resolved, but you may open a separate feature request for MS-SSIM if desired.

[grlee77]

Nevermind, I have reopened again as it looks like pytorch-msssim can return both MS-SSIM and SSIM values and you seem to be correctly comparing SSIM. Still, I would try comparing to other implementations such as the PyTorch Ignite one to see if it also differs from those.

If your image is very small, one other source of potential difference is from boundary pixels. Our implementation discard a small boundary at the image edge when computing the mean, which for images of say 256x256, will not make a large difference, but for tiny images such as 16x16 might make a substantial difference in the computed value.

[grlee77]

As another test, you could try running this test case, but using pytorch-msssim:

scikit-image/skimage/metrics/tests/test_structural_similarity.py

Lines 193 to 214 in 43d79b8

	def test_gaussian_structural_similarity_vs_IPOL():
	""" Tests vs. imdiff result from the following IPOL article and code:
	https://www.ipol.im/pub/art/2011/g_lmii/.
	Notes
	-----
	To generate mssim_IPOL, we need a local copy of cam_noisy:
	>>> from skimage import io
	>>> io.imsave('/tmp/cam_noisy.png', cam_noisy)
	Then, we use the following command:
	$ ./imdiff -m mssim <path to camera.png>/camera.png /tmp/cam_noisy.png
	Values for current data.camera() calculated by Gregory Lee on Sep, 2020.
	Available at:
	https://github.com/scikit-image/scikit-image/pull/4913#issuecomment-700653165
	"""
	mssim_IPOL = 0.357959091663361
	mssim = structural_similarity(cam, cam_noisy, gaussian_weights=True,
	use_sample_covariance=False)
	assert_almost_equal(mssim, mssim_IPOL, decimal=3)

The original author's MATLAB script also gave the same value to a few digits, but I think we dropped that test case as redundant when we updated our cameraman image a year or so ago.

[Pixie8888]

Nevermind, I have reopened again as it looks like pytorch-msssim can return both MS-SSIM and SSIM values and you seem to be correctly comparing SSIM. Still, I would try comparing to other implementations such as the PyTorch Ignite one to see if it also differs from those.

If your image is very small, one other source of potential difference is from boundary pixels. Our implementation discard a small boundary at the image edge when computing the mean, which for images of say 256x256, will not make a large difference, but for tiny images such as 16x16 might make a substantial difference in the computed value.

Hi,
Yes. My images are rather small: 64*64, value range (0~1). I use pytorch_mssim.ssim to compute ssim (https://pypi.org/project/pytorch-msssim/), pytorch version 1.4.

[Pixie8888]

Hi, I upload complete code here:

import torch
import numpy as np
import os
from pytorch_msssim import ssim as ssim_func
from skimage.measure import compare_psnr as psnr_metric
from skimage.measure import compare_ssim as ssim_metric
from PIL import Image

def eval_seq_skimage(gt, pred):
    T = len(gt)
    ssim = np.zeros(T)
    for t in range(T):
        ssim[t] = ssim_metric(gt[t], pred[t], sigma=1.5, use_sample_covariance=False, gaussian_weights=True, multichannel=True) # (h, w,3)

    return ssim

def eval_ssim_pytorch(gt, pred):
    # (n, 3, h, w)
    T = len(gt)
    ssim = torch.zeros(T)
    for t in range(T):
        ssim[t] = ssim_func(torch.tensor(gt[t]).permute(2,0,1).unsqueeze(0), torch.tensor(pred[t]).permute(2,0,1).unsqueeze(0), data_range=1, size_average=False)[0] # (1,3,h,w)
    return ssim


pred_path = 'attention_compare/NewUnet_Perceptual_AttentionLoss/3432_pred'
gt_path = 'attention_compare/NewUnet_Perceptual_AttentionLoss/3432_gt'
gt_list = []
pred_list = []
for i in range(len(os.listdir(pred_path))):
    pred_img = Image.open(os.path.join(pred_path, str(i) + '.jpg'))  ##64*64
    pred_img = np.array(pred_img)  ##64,64,3
    pred_img = pred_img / 255.0  ##(0, 1)  
    pred_list.append(pred_img)

    gt_img = Image.open(os.path.join(gt_path, str(i) + '.jpg'))
    gt_img = np.array(gt_img)
    gt_img = gt_img / 255.0 # (h, w, 3)
    gt_list.append(gt_img)

ssim_skimage = eval_seq_skimage(gt_list, pred_list)
ssim_pytorch = eval_ssim_pytorch(gt_list, pred_list)
print(ssim_skimage[-30:])
print(ssim_pytorch[-30:])

Results:
`from skimage: [0.98778352 0.97618806 0.94805653 0.91541467 0.89592568 0.87270792
0.84890676 0.82580515 0.80778026 0.79459803 0.77772534 0.77228931
0.77941197 0.79023229 0.81249131 0.85597647 0.88561149 0.86743074
0.8334235 0.80246721 0.78719885 0.7763473 0.7854198 0.80604551
0.81067763 0.81544572 0.81859287 0.82002742 0.81960349 0.81605346]

from pytorch_mssim.ssim: tensor([0.9836, 0.9678, 0.9323, 0.8911, 0.8668, 0.8392, 0.8136, 0.7872, 0.7673,
0.7532, 0.7345, 0.7278, 0.7338, 0.7439, 0.7659, 0.8125, 0.8449, 0.8216,
0.7829, 0.7483, 0.7312, 0.7191, 0.7299, 0.7547, 0.7604, 0.7664, 0.7700,
0.7725, 0.7729, 0.7697])`

[rfezzani]

My two cents: isn't it a dtype problem?

>>> np.zeros(5).dtype
dtype('float64')
>>> torch.zeros(5).dtype
torch.float32

[ChenBinfighting1]

Can I assign it？

[grlee77]

Comments moved over from the Discussion

(we are planning to disable Discussions again and just keep the issues page)

rfezzani:

@Pixie412 , I see that pytorch-msssim provides a consistency test with skimage: https://github.com/VainF/pytorch-msssim/blob/master/tests/tests_comparisons_skimage.py
Can you please run this test?

Pixie412:

The parameters used in this test are
https://github.com/VainF/pytorch-msssim/blob/6ceec02d64447216881423dd7428b68a2ad0905f/tests/tests_comparisons_skimage.py#L36-L37
for skimage and
https://github.com/VainF/pytorch-msssim/blob/6ceec02d64447216881423dd7428b68a2ad0905f/tests/tests_comparisons_skimage.py#L48
for pytorch-msssim.

rfezzani:

@Pixie412, I updated your code to use the parameters defined in my previous message. Can you try this?

import torch
import numpy as np
import os
from pytorch_msssim import ssim as ssim_func
from skimage.metrics import peak_signal_noise_ratio as psnr_metric
from skimage.metrics import structural_similarity as ssim_metric
from PIL import Image

def eval_seq_skimage(gt, pred):
    T = len(gt)
    ssim = np.zeros(T)
    for t in range(T):
        ssim[t] = ssim_metric(gt[t], pred[t], win_size=11, sigma=1.5,
                              use_sample_covariance=False, gaussian_weights=True,
                              multichannel=True, data_range=255) # (h, w,3)

    return ssim

def eval_ssim_pytorch(gt, pred):
    # (n, 3, h, w)
    T = len(gt)
    ssim = torch.zeros(T)
    for t in range(T):
        ssim[t] = ssim_func(torch.tensor(gt[t]).permute(2,0,1).unsqueeze(0),
                            torch.tensor(pred[t]).permute(2,0,1).unsqueeze(0),
                            data_range=255, win_size=11) # (1,3,h,w)
    return ssim


pred_path = 'attention_compare/NewUnet_Perceptual_AttentionLoss/3432_pred'
gt_path = 'attention_compare/NewUnet_Perceptual_AttentionLoss/3432_gt'
gt_list = []
pred_list = []
for i in range(len(os.listdir(pred_path))):
    pred_img = Image.open(os.path.join(pred_path, str(i) + '.jpg'))  ##64*64
    pred_img = np.array(pred_img)  ##64,64,3
    pred_img = pred_img / 255.0  ##(0, 1)  
    pred_list.append(pred_img)

    gt_img = Image.open(os.path.join(gt_path, str(i) + '.jpg'))
    gt_img = np.array(gt_img)
    gt_img = gt_img / 255.0 # (h, w, 3)
    gt_list.append(gt_img)

ssim_skimage = eval_seq_skimage(gt_list, pred_list)
ssim_pytorch = eval_ssim_pytorch(gt_list, pred_list)
print(ssim_skimage[-30:])
print(ssim_pytorch[-30:])

[grlee77]

I moved the discussion back to the issues page. If there is further feedback we can reopen the issue