pre-trained student weights

[roymiles]

Hi
Thanks for releasing this project! You mentioned it is beneficial to initialize the student with pre-trained weights. Do you provide these weights by any chance? Specifically for the models used for the SQuAD task (L3 etc)

At the moment we are just copying the pre-trained weights from base bert into the first few 3 layers, but I am unsure if this is the best/recommended way of doing it.

Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[airaria]

Hi Thanks for releasing this project! You mentioned it is beneficial to initialize the student with pre-trained weights. Do you provide these weights by any chance? Specifically for the models used for the SQuAD task (L3 etc)

At the moment we are just copying the pre-trained weights from base bert into the first few 3 layers, but I am unsure if this is the best/recommended way of doing it.

Thanks

We initialized the student with the first 3 layers of BERT, same as what you did.
It is the recommended way (I have experimented other schemes and I don't know if there are other simple and better ways)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.