New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bert.pooler.dense initialization #300

Closed

SinghJasdeep opened this issue Feb 20, 2019 · 1 comment

Contributor

SinghJasdeep commented Feb 20, 2019

Hey! Sorry if this is redundant.

I saw 3 other issues asking similar questions but couldn't find these exact layers mentioned.

Are bert.pooler.dense.weight & bert.pooler.dense.bias randomly initialized?

Thank you so much!

Member

thomwolf commented Feb 20, 2019

Hi Jasdeep,
No, they are initialized from Google's pretrained model (they are trained for next sentence prediction task during pretraining).

thomwolf closed this as completed

hmohebbi mentioned this issue

Why LayerNorm layers are frozen? hmohebbi/TF-Adapter-BERT#1

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment