-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why the style encoder maps all the images to the same style code? #28
Comments
It might be ... |
|
@FriedRonaldo BTW the quality of the generated samples at inference are great they only lack the style 😄 |
After BTW the quality of the generated samples at inference are great they only lack the style The meaning of "if the results at the very last iteration look good" is not just about the quality but how well the samples reflect the style images. If you use a black cat as a style image, then, is the output a black cat? If yes in the training and if not in the inference, more information should be provided to diagnose the issue. Blind diagnosis is somewhat difficult... |
Sorry for causing some confusions. If you use a black cat as a style image, then, is the output a black cat? I did an experiment where I overfit the model during testing i.e. gave the model a seen content and seen style image which it perfectly generated during training, but surprisingly it failed during inference as well i.e. style code doesn't match the style image however the content is fine. If yes in the training and if not in the inference, more information should be provided to diagnose the issue. Blind diagnosis is somewhat difficult |
In this case, the problem is just about the inference, so using the style contrastive loss does not solve the problem. In my experience, there is an issue in the batch normalization layer and if I use "eval()", the model does not work, but without it, the model works well. It might be related to the normalization layer. Or EMA might raise the problems. I recommand to use the generator that is non-EMA version. Because it is not related to our source code, I am sorry for that I can not help you more. |
@FriedRonaldo Thank you for helping me this far and giving some ideas. I will try with your feedback hopefully I can resolve it. |
Hello. Could you please give some explanation and advise for the scenario explained below.
I am not using TUNIT but a very similar architecture where I have a GAN jointly trained with a style encoder. The style encoder is trained as a classifier in a supervised setting and its middle layer features are injected as a style code in the generator. During training the validation results are pretty good (which I assume is down to the L1 loss I am using because of the paired supervision).
During inference the style encoder maps all images to a same style code.
The exact same scenario is mentioned in section 3.3. of your paper where you define the style contrastive loss for the generator. Quoting from your paper "This loss guides the generated image G(x, ˜s) to have a style similar to the reference
image x˜ and dissimilar to negative (other) samples. By doing so, we avoid the degenerated solution where the encoder
maps all the images to the same style code of the reconstruction loss [5] based on L1 or L2 norm." Where reference [5] is the starGAN-v2.
In my case I am also using a style classification loss on the style image for the generator however it seems to completely ignore it during inference.
Could you please explain
The text was updated successfully, but these errors were encountered: