LeakyReLU after Conv or WScale?

[taoari]

From https://github.com/ptrblck/prog_gans_pytorch_inference/blob/master/network.py#L470, LeakyReLU is inserted after Conv. However, from https://github.com/ptrblck/prog_gans_pytorch_inference/blob/master/model.py#L52, LeakyReLU is inserted after WScale layer. So should LeakyReLU be inserted after Conv or WScale? You have checked the output in the transfer_weights.py script, did I miss something?

BTW, do you also have the script for the discriminator network? I am trying to implement the training of the "Progressive Growing of GANs" paper in PyTorch based on your script. Looking forward to hearing back from you.

[ptrblck]

You are mostly right, but have a look at the behavior of the WScale layer: https://github.com/ptrblck/prog_gans_pytorch_inference/blob/master/network.py#L292
It is basically "stealing" the nonlinearity from its incoming layer und resets it to linear, if the incoming layer has a nonlinearity set.

That's why I just placed the LeakyReLU after the WScale layer, since all Conv layer in the Lasagne net have a LeakyReLU set.

Unfortunately, I don't have the discriminator code, but I would love to implement it in the next iteration.
Have you already started implementing it?

[taoari]

@ptrblck Thank you! I have the base code for GAN training, and start to figure output the discriminator code.