Does it make sense to do spectral normalization before batch normalization?

[richardwth]

Would the spectral norm get canceled out because it appears on both the nominator and denominator of the batch normalization equation?

I mean:
bn(x*w/sn(w)) = gamma * (x*w/sn(w) - mean(x*w/sn(w))) / std(x*w/sn(w)) + beta = bn(x*w)

[krishnatoshniwal]

@richardwth Did you end up experimenting with any other order?

[lzw-all-in]

but i think
std(xw/sn(w)) = std(xw) / (sn(w)^2)

[richardwth]

but i think
std(xw/sn(w)) = std(xw) / (sn(w)^2)

No... std(x * c) = std(x) * c.

[jerofad]

And why was the 2 used even at the same time. I dont seem to understand why Spectral Norm and BatchNorm are used together. Can someone explain?

[mcstarioni]

They are orthogonal by their effect. SN changes weights of the layer, BN changes activations.

[15732031137]

Hello! Thank you for your contribution to generative adversarial network research and for sharing your code! I am from China. Now is the Chinese New Year. I wish you a happy Chinese New Year! I am very interested in your thesis, when I try to add spectral normalization in my new networks, the program gives the following error:
Traceback (most recent call last):
File "SR.py", line 45, in
train(opt, Gs, Zs, reals, NoiseAmp)
File "E:\SinGAN-masterplus\SinGAN\training.py", line 34, in train
D_curr,G_curr = init_models(opt)
File "E:\SinGAN-masterplus\SinGAN\training.py", line 310, in init_models
netG.apply(models.weights_init)
File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 293, in apply
module.apply(fn)
File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 293, in apply
module.apply(fn)
File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 294, in apply
fn(self)
File "E:\SinGAN-masterplus\SinGAN\models.py", line 215, in weights_init
m.weight.data.normal_(0.0, 0.02)
File "E:\abcd\lib\site-packages\torch\nn\modules\module.py", line 591, in getattr
type(self).name, name))
AttributeError: 'Conv2d' object has no attribute 'weight'

I have searched a lot of information and couldn't solve it, so I want to ask you, I wish you a happy life, and look forward to your reply!