Error while calculating FID score for a small dataset

[sagarkora]

I'm calculating FID score for a small dataset - folder1 - 1341 images and folder 2 - 1000 images with parameters as --dims 768. Please see below for the error-

100%|██████████| 10/10 [00:11<00:00, 1.11s/it]Warning: batch size is bigger than the data size. Setting batch size to data size

---

ValueError Traceback (most recent call last)
/usr/lib/python3.6/runpy.py in run_module(mod_name, init_globals, run_name, alter_sys)
203 run_name = mod_name
204 if alter_sys:
--> 205 return _run_module_code(code, init_globals, run_name, mod_spec)
206 else:
207 # Leave the sys module alone

7 frames
/content/drive/My Drive/pytorch-fid-master/pytorch_fid/fid_score.py in get_activations(files, model, batch_size, dims, cuda)
100 pred_arr = np.empty((len(files), dims))
101
--> 102 for i in tqdm(range(0, len(files), batch_size)):
103 start = i
104 end = i + batch_size

ValueError: range() arg 3 must not be zero
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2590: UserWarning: Unknown failure executing module: <pytorch_fid>
warn('Unknown failure executing module: <%s>' % mod_name)

[mseitzer]

Are you sure that there any images in the folders you are using? It recognizes only images with extensions .jpg and .png (small caps).

[sagarkora]

Ok. I missed that part. I changed the extension from .jpeg to .png and ran this again with the below error. As mentioned in Support grayscale images #41 - I did update the code to convert to RGB, but still, I ran into the below error:

0%| | 0/20 [00:00<?, ?it/s]

5%|▌ | 1/20 [00:00<00:06, 3.15it/s]

10%|█ | 2/20 [00:00<00:05, 3.32it/s]

15%|█▌ | 3/20 [00:00<00:04, 3.42it/s]

20%|██ | 4/20 [00:01<00:04, 3.51it/s]

25%|██▌ | 5/20 [00:01<00:04, 3.60it/s]

30%|███ | 6/20 [00:01<00:03, 3.62it/s]

35%|███▌ | 7/20 [00:01<00:03, 3.70it/s]

40%|████ | 8/20 [00:02<00:03, 3.72it/s]

45%|████▌ | 9/20 [00:02<00:02, 3.75it/s]

50%|█████ | 10/20 [00:02<00:02, 3.74it/s]

55%|█████▌ | 11/20 [00:02<00:02, 3.71it/s]

60%|██████ | 12/20 [00:03<00:02, 3.74it/s]

65%|██████▌ | 13/20 [00:03<00:01, 3.75it/s]

70%|███████ | 14/20 [00:03<00:01, 3.76it/s]

75%|███████▌ | 15/20 [00:04<00:01, 3.73it/s]

80%|████████ | 16/20 [00:04<00:01, 3.75it/s]

85%|████████▌ | 17/20 [00:04<00:00, 3.77it/s]

90%|█████████ | 18/20 [00:04<00:00, 3.79it/s]

95%|█████████▌| 19/20 [00:05<00:00, 3.79it/s]

100%|██████████| 20/20 [00:05<00:00, 3.73it/s]

0%| | 0/20 [00:01<?, ?it/s]

ValueError Traceback (most recent call last)
/usr/lib/python3.6/runpy.py in run_module(mod_name, init_globals, run_name, alter_sys)
203 run_name = mod_name
204 if alter_sys:
--> 205 return _run_module_code(code, init_globals, run_name, mod_spec)
206 else:
207 # Leave the sys module alone

7 frames
/content/drive/My Drive/pytorch-fid-master/pytorch_fid/fid_score.py in get_activations(files, model, batch_size, dims, cuda)
108
109 # Reshape to (n_images, 3, height, width)
--> 110 images = images.transpose((0, 3, 1, 2))
111 images /= 255
112

ValueError: axes don't match array
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2590: UserWarning: Unknown failure executing module: <pytorch_fid>
warn('Unknown failure executing module: <%s>' % mod_name)

[mseitzer]

There still seems to be something wrong with your images, or how they are loaded to arrays. Namely it appears as if images does not have 4 dimensions at this point. Can you insert print(images.shape) before the transpose line and post the result?

[sagarkora]

print(images.shape)
(50,)

[sagarkora]

Ok, I guess the size of the image varies. After adding the resize(299,299)- (to match the input of Inception model) to the line where images load solves the problem. Is there a better solution? After changing the code, I calculated FID score using --dims 768 which resulted in the FID score of 1.3

[mseitzer]

It is intentional that images are not automatically resized to the same size because there are several ways you can get images of different size to the same size (e.g. cropping, rescaling) and it is up to the user to decide which way works best for the used dataset.

Regarding the score you get, keep in mind that normal FID score uses the full 2048 dimensions. So you can not compare scores with 768 dimensions to the normal FID score, and they might behave entirely different. They might not even reflect human judgment of image quality.

[sagarkora]

I understand and agree with your comment. But, do you think we can still calculate FID score using 2048 dimensions for the samples less than 2048 in either of the two comparable folders(images). I know we might not have the full rank and covariance matrix and some NAN's. But, I'm just wondering if it makes sense to even give it a try?

[mseitzer]

It depends on what you do with the results. Maybe you would be able to compute an approximation to FID using less than 2048 samples, but it's hard to predict how good the quality of this approximation is. You surely can not compare against FID scores computed on full 2048 samples.