Got wrong metric results

[TrickyGo]

Help, please! I used skimage.measure.compare_ssim to your CelebA-HQ_pretrained on the test set and got a result SSIM of 0.54 which was supposed to be 0.73 as in your paper. The generated image looks just fine with high reconstruction quality. Would you please tell me where has gone wrong? (actually other metrics went wrong too. https://github.com/mseitzer/pytorch-fid got a FID of 30 but was supposed to be 17.66)
Here's my code:

def ssim_score(generated_images, reference_images):
    ssim_score_list = []
    for reference_image, generated_image in zip(reference_images, generated_images):

        ssim = skimage.measure.compare_ssim(reference_image, generated_image, multichannel=True,
                            data_range=generated_image.max() - generated_image.min())
        ssim_score_list.append(ssim)
    return np.mean(ssim_score_list)

[TrickyGo]

Since the original image is (1024,1024,3) different from generated image, I did the resize as

skimage.transform.resize(reference_image, (256, 256), 3, preserve_range=True, anti_aliasing=False)

Anything wrong?

[ZPdesu]

For the SSIM, first you need to normalize the images into range 0-1, second use the default data_range (estimated from the image data-type.) in the skimage.measure.compare_ssim function.

image1 = skimage.io.imread('test1.png')/255
image2 = skimage.io.imread('generated1.png')/255
compare_ssim(image1, image2, multichannel=True, gaussian_weights=True, use_sample_covariance=False)

For the FID, you should calculate the FID between training set (28000) and the generated images (2000).

calculate_fid_given_paths(paths=[Train_folder, Gen_folder], batch_size=50, cuda=True, dims=2048)

The results will be slightly different from what reported in the paper because of the slight architecture difference and training randomness.

[TrickyGo]

Thanks you, really appreciate it! I didn't nomalize before and I thought FID was between test and generated. Now the SSIM turns to resonable 0.7 but FID still above 30, here's what's in my terminal, is it normal?

paths: ['SEAN-master/datasets/CelebA-HQ/train/images', 
'SEAN-master/results/CelebA-HQ_pretrained/test_latest/images/synthesized_image']
InceptionV3 model defined
100%|█████████████████████████████████████████| 560/560 [16:40<00:00,  1.74s/it]

 m1, s1: [0.22765248 0.19109708 0.36054623 ... 0.31543327 0.17475582 0.46798471]                                                                                                              [[ 0.03107088  0.00358307 -0.00253577 ...  0.00283327  0.0020783
  -0.00186917]
 [ 0.00358307  0.02236766  0.00087384 ...  0.00334818  0.00164374
  -0.00246213]
 [-0.00253577  0.00087384  0.03687847 ...  0.00151311  0.00311874
   0.00234653]
 ...
 [ 0.00283327  0.00334818  0.00151311 ...  0.05386037  0.0014196
   0.00197556]
 [ 0.0020783   0.00164374  0.00311874 ...  0.0014196   0.02679679
   0.00170726]
 [-0.00186917 -0.00246213  0.00234653 ...  0.00197556  0.00170726
   0.06954509]]
100%|███████████████████████████████████████████| 40/40 [00:10<00:00,  3.66it/s]

 m2, s2: [0.36204847 0.17871976 0.28304769 ... 0.31455503 0.18665536 0.46938161]                                                                                                              [[ 0.05629672  0.00130017 -0.00044781 ...  0.00708514  0.00631912
   0.00524824]
 [ 0.00130017  0.02214662 -0.00021558 ...  0.00468644  0.00081381
  -0.00371283]
 [-0.00044781 -0.00021558  0.02702752 ...  0.00294189  0.00157947
   0.00125132]
 ...
 [ 0.00708514  0.00468644  0.00294189 ...  0.05291733  0.00327614
   0.00449331]
 [ 0.00631912  0.00081381  0.00157947 ...  0.00327614  0.02777803
   0.00402917]
 [ 0.00524824 -0.00371283  0.00125132 ...  0.00449331  0.00402917
   0.07106463]]
FID:  31.974111731293704

[ZPdesu]

Did you save your resized training images as png? If they were saved as jpg, the results would definitely drop a lot.

[TrickyGo]

OK. Thank you!