-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lm_head and v_head, why re-initialize and why dropout? #43
Comments
So I did some research on my own and basically my first 2 questions can be answered by looking at the huggingface transformers repository: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py |
Hi @clam004, do you mind explaining your answer/understanding on why they do it? Thanks! |
@danjohnvelasco as long as you use the same name |
Regarding 3 I agree and we moved the dropout before the linear layer in #70. |
First off, thank you for building this! 3 questions regarding the two heads of the policy model:
when a trained lm_head already exist in GPT2LMHeadModel?
from 01-gpt2-with-value-head.ipynb
Thanks again!
The text was updated successfully, but these errors were encountered: