Cannot reproduce the FID results of provided pre-trained models (LSUN - Churches & CelebA)

[hieuphung97]

Has anyone had the issue when re-evaluating the provided pre-trained models on LSUN - Churches and CelebA datasets?
I cannot get the FIDs as reported in the paper.
In fact, the results are far from the reported ones (LSUN-Churches: 4.02 in the paper compared with 11.5, CelebA: 5.11 in the paper compared with 17.4)

The only difference I noticed is that I sampled only 10k images for each case instead of 50k as in the paper (to save time). I don't know whether the number of samples has this significant impact.

[dongzhuoyao]

I think larger sampling number like 50k can further drop the performance by 30%?

[ader47]

Hi @hieuphung97, have you reproduced the results reported in the paper? I have sampled 35k images using the provided pre-trained model on lsun_churches, calculated the FID via the torch-fidelity, and the FID was 15.89.

[ThisisBillhe]

Hi @hieuphung97 , I wonder if you are using the celebA or celebA-HQ dataset? I got a FID of 27 on celebA-HQ dataset.

[ader47]

Hi @hieuphung97 , I wonder if you are using the celebA or celebA-HQ dataset? I got a FID of 27 on celebA-HQ dataset.

Does your dataset for caculating FID pre-processed correctly? It's quite important.

[ThisisBillhe]

Hi @hieuphung97 , I wonder if you are using the celebA or celebA-HQ dataset? I got a FID of 27 on celebA-HQ dataset.

Does your dataset for caculating FID pre-processed correctly? It's quite important.

I use pytorch-fid, which directly uses path-to-dataset-folder as args... So it should not be a pre-processing issue

[ader47]

Are the images in the dataset size 256*256?

[ThisisBillhe]

Are the images in the dataset size 256*256?

Yes...I download the celebA-HQ dataset from here. But I would greatly appreciate it if you could provide me with the URL of your dataset. I would have a try.

[ThisisBillhe]

Hi @hieuphung97, have you reproduced the results reported in the paper? I have sampled 35k images using the provided pre-trained model on lsun_churches, calculated the FID via the torch-fidelity, and the FID was 15.89.

Update: I also got a FID of ~17 on CelebA dataset, which is similar to @hieuphung97

[hieuphung97]

Hi @ader47 @ThisisBillhe
The issue is solved by using the correct inference config and the correct number of generated images

• DDIM step: 200
• eta: 0
• Num of samples: 50k

@ThisisBillhe I use celebA-HQ dataset

[ader47]

Thank you so much :)

[Alii-Ganjj]

Hi @ader47 @ThisisBillhe The issue is solved by using the correct inference config and the correct number of generated images

• DDIM step: 200
• eta: 0
• Num of samples: 50k

@ThisisBillhe I use celebA-HQ dataset

Hi @hieuphung97
I am trying to reproduce the results for LSUN-Church. I have used the same setting that you mentioned, but I got FID 17.58 which is far from 4.02. How could you solve this issue?

Details of what I did: I sampled 50k images with the setting that you mentioned. Then, I wrote a script that instantiates two dataloaders with the same pre-processing as the original ones for the real and fake image directories. I pass these two dataloaders to the torch-fidelity package as the paper suggested.