You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In #10 we observed that it looks like a lot of the weights in the embeddings are never seen during training so maintain their initialized values. From what we can tell this seems to be happening for most weights initialized with a negative value. If we randomly initialize our embedding layer, weights that have never been seen during training have little to contribute to prediction at test time as their value is random. We may be able to use this to make our saved models smaller.
After training, we could check which weights have values that have not changed from their initialized value and set them to zero. Then when saving the matrix of embedding weights we just need to save the non-zero values and where they occur in the matrix.
The text was updated successfully, but these errors were encountered:
In #10 we observed that it looks like a lot of the weights in the embeddings are never seen during training so maintain their initialized values. From what we can tell this seems to be happening for most weights initialized with a negative value. If we randomly initialize our embedding layer, weights that have never been seen during training have little to contribute to prediction at test time as their value is random. We may be able to use this to make our saved models smaller.
After training, we could check which weights have values that have not changed from their initialized value and set them to zero. Then when saving the matrix of embedding weights we just need to save the non-zero values and where they occur in the matrix.
The text was updated successfully, but these errors were encountered: