Gradient Penalty is very high in the start

[inspirit]

Hi!

i was running few experiments and noticed that GP is extremely hight in first few 100 steps.
GP > 60000, and then gradually going down to around GP = 20

is it normal behaviour? In my previous experience with StyleGan GP was small in the beginning

[lucidrains]

@inspirit hey, so i also noticed that. however, i believe when i turned off the multi-scale (one of the main contributions of the paper), the gradients resembled stylegan, so i attributed it to that

did you close the issue because your training was successful?

[inspirit]

hey! i closed it because the results were fine, but i did not train it for long, i'm borrowing different ideas from various GAN implementations for some experiments. GigaGan discriminator is definitely not the most stable and D/MSD/GP are fluctuating a lot

[lucidrains]

@inspirit yes, i think it is the multi-scale causing it, last i was training on my toy datasets. if you turn it off by setting the loss weight to 0, the training resembles stylegan a lot (could be remembering wrong)

[inspirit]

so whats the main purpose of having Multi Scale Discriminator in this case?

[lucidrains]

@inspirit i think it is necessary to achieve the paper's results

[lucidrains]

@inspirit are you a phd student btw? we run into each other a lot

[inspirit]

Haha I'm long time past that age i believe :) Just freelance researcher and technical adviser for CV/ML stuff

[lucidrains]

@inspirit ah got it :) well, glad to have you reviewing my code from time to time!

[inspirit]

@inspirit i think it is necessary to achieve the paper's results

in my opinion adaptive mod-conv is more useful and can be fitted in lots of ideas...

[lucidrains]

@inspirit yea i think both may be important