Learn diversity

[remega]

Hi Junyan,

Thanks for your impressive work. Recently, I apply BicycleGAN-like framework into my GAN model, hoping it learn to generate diverse results. However, my model seems not sensitve to the latent code at all. The outputs look almost same with differnt latent code, both over training and validation stages, even I enlarge the weights of KL-loss and l1-loss on latent code. My KL_loss is becoming very small during training, but the l1-loss can only be converaged to around 0.7. Do you remember what are converaged values of these losses during your training? And kindly do you have any idea or advice of it? Thanks you so much！

Looking forward to your reply.

Best,
Lai

[junyanz]

If you implement your own model with the BicycleGAN style, I will first test the model without KL loss and L1-loss on z. And then add KL loss and see if it works, and finally add L1-loss. The hyper-parameters in your model might be different from the ones used in our paper. Notably, how to inject noises into the generator is also important. May you want to inject it into multiple layers, or using more recent architecture such as SPADE.

[remega]

Thank you a lot for your advices. Actually, my model can generate real-style outputs, whether adding l1-loss and kl loss. But the model seems to always overlook the latent codes. I have tried add the latent code on every layer of Generator, and I also tried different loss weights. All failed. Do you think there is any other adjustments I can do for highlighting the learning of latent codes? I notice that in your paper you said if E and G were trained simultaneously, the information of latent codes are trend to be hidden, so you only update G via z l1-loss. Do you think if I update EG and G in different frequency would help?

I would appreciate so much if you can give me some advices.

[junyanz]

Do you have a reconstruction loss on the image ||G(x, E(y)) - y||?

[remega]

Yes, I have.

[junyanz]

I see. I think only updating G helps. I recommend that you first start from pix2pix, and gradually add more losses.

[remega]

Thanks a lot for your reply. Actually, I verify my model (and the new task) on a cycleGAN style GAN, to get the realistic output through adjusting the structures and hyper-parameters. But how can I verify if cycleGAN/Pix2Pix style model is able to generate the diversity outputs? Do you have any suggestions on it? Thank you so much.

[junyanz]

To quantify the output diversity, you can use the LPIPS distance. See Sec 5.2 in the BicycleGAN paper for more details.

[remega]

Okay, your advice really helps. Thanks a lot.

[soon-yau]

I'm also facing same problem. I implemented BicycleGAN from a version of my pix2pix that works. Then I train my BicycleGAN on facades dataset but it is still stuck at one style. If I over train it, then it ends up with mode collapse than the generator is only producing same image regardless of the input. I wonder if the dataset is no good for this as it is not included in BicycleGAN paper, unlike previously with pix2pix and CyleGan.