layernorm should be set to trainable or not ?

[rabeehkarimimahabadi]

Hi,
I see in some implementations they set layer_norm as require_grad=True, could you tell me if all layer norms of the model needs to be set to require_grad=True, or only the ones inside adapter layer needs this condition?
thanks.

[arueckle]

See figure 3 in the adapterhub paper, everything outside the pink box is fixed/frozen during adapter training

[rabeehkarimimahabadi]

Hi Andreas but looking into their original paper implementation, this is not frozen, also classifier is not. Could you confirm which one is correct? thanks

…

On Sun, Nov 29, 2020 at 1:18 PM Andreas Rücklé ***@***.***> wrote: See figure 3 in the adapterhub paper, everything outside the pink box is fixed/frozen during adapter training — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#98 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARPXHHYWQ3VCTZNDMC4WVEDSSI3YXANCNFSM4UGPBTOQ> .