Question on clamp

[lizhenstat]

Hi, I have a question on clap weights
https://github.com/ShichenLiu/CondenseNet/blob/master/layers.py#L125

weight = weight.sum(0).clamp(min=1e-6).sqrt()

I don't understand the clamp function here. I tried to train condensenet-86 on cifar10 . with and without clamp functions
with clamp: error rate = 95.06
without clamp: error rate = 94.96

Thanks in advance

[ShichenLiu]

Hi,

The clamp function here is to prevent numerical issues. Since the derivative of x^p is p*x^(p-1), when p < 1, it is essential to prevent x^(p-1) to be too big which introduces numerical unstable issues.

[lizhenstat]

@ShichenLiu Oh! I got it, thanks a lot