New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FID increases for larger training datasets #55
Comments
"While the training FID-5K score is lower when training on 3.8K vs 750" What does this sentence mean? Could you give more details? |
When training on 3.8K images, the fid-5k score monitored through training
converges to around 8.0. When training on 750 images, the fid-5k score
monitored through training converges to around 33.
When I evaluate the fid-50k score for the generative model trained on 750
images, it has an fid-50k score around 21. When I evaluate the fid-50k
score the generative model trained on 3.8K images, it has an fid-50k score
around 50.
Best,
Sayeri Lala
PhD candidate | Electrical Engineering | Princeton University
…On Fri, Mar 12, 2021 at 8:31 PM Shengyu Zhao ***@***.***> wrote:
"While the training FID-5K score is lower when training on 3.8K vs 750"
What does this sentence mean? Could you give more details?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#55 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AN3GCFSL32DL75IQ4OJNM4LTDLTCNANCNFSM4ZDDMJPA>
.
|
This is probably caused by the inconsistency between training and testing. The FID-50k score of a model should be smaller than FID-5k of the same model. Please check whether your data is properly processed during testing. |
By inconsistency, do you mean with respect to image preprocessing?
Best,
Sayeri Lala
PhD candidate | Electrical Engineering | Princeton University
…On Fri, Mar 12, 2021 at 11:32 PM Shengyu Zhao ***@***.***> wrote:
This is probably caused by the inconsistency between training and testing.
The FID-50k score of a model should be smaller than FID-5k of the same
model.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#55 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AN3GCFQKBQLGFH2DE3WSNNDTDMIJJANCNFSM4ZDDMJPA>
.
|
Or are there other differences bet training and testing for the generator
e.g., dropout, batch norm? Thanks.
Best,
Sayeri Lala
PhD candidate | Electrical Engineering | Princeton University
…On Fri, Mar 12, 2021 at 11:45 PM Sayeri Lala ***@***.***> wrote:
By inconsistency, do you mean with respect to image preprocessing?
Best,
Sayeri Lala
PhD candidate | Electrical Engineering | Princeton University
On Fri, Mar 12, 2021 at 11:32 PM Shengyu Zhao ***@***.***>
wrote:
> This is probably caused by the inconsistency between training and
> testing. The FID-50k score of a model should be smaller than FID-5k of the
> same model.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#55 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN3GCFQKBQLGFH2DE3WSNNDTDMIJJANCNFSM4ZDDMJPA>
> .
>
|
Most likely due to image preprocessing. |
Another guess - maybe you computed the FID statistics of real images with 750 images, but when you evaluate the model trained with 3.8k images, it still uses the cached statictics with 750 images. Maybe you can clear the cached files or rename the dataset, and then re-evaluate the model trained with 3.8k images. |
Thanks, I'll look into it.
Best,
Sayeri Lala
PhD candidate | Electrical Engineering | Princeton University
…On Sat, Mar 13, 2021 at 3:37 AM Shengyu Zhao ***@***.***> wrote:
Another guess - maybe you computed the FID statistics of real images with
750 images, but when you evaluate the model trained with 3.8k images, it
still uses the cached statictics with 750 images. Maybe you can clear the
cached files or rename the dataset, and then re-evaluate the model trained
with 3.8k images.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#55 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AN3GCFSZL5HQZ5Y745347GLTDNFABANCNFSM4ZDDMJPA>
.
|
I made some fixes so now the results match. Thanks for your thoughts! |
I trained a generator using 3.8K images on 256x256 following the settings described and trained another generator using 750 images with the same settings but with regularization increased from 1 to 10 as in the paper.
While the training FID-5K score is lower when training on 3.8K vs 750 (both training FID curves are stable), I find that the the FID score computed using the generator trained on 3.8K gives much higher FID scores compared to that attained under 750.
I am using the same code to compute the FID scores so it is unclear to me why this is the case. Have you encountered this issue?
The text was updated successfully, but these errors were encountered: