Train from scratch

[stomachacheGE]

Thanks for your work.

I have a question concerning training from scratch. I checked the source code, and it seems that there is no implementation of position embedding. One can only load position embedding from the pretrained models. If I want to train from scratch, should I implement position embedding by myself, or is there something I overlooked? Any other things I should be careful with if training from scratch?

[jeonsworld]

Training from scratch is the same as position embedding in the source code, the difference is that it doesn't initialize with pre-train weights.
Referring to the author's code, the author initializes the weights with the normal distribution (std=0.02).

The pytorch implementation is as follows:

position_embeddings = nn.Parameter(torch.empty(1, n_patches+1, hidden_size), requires_grad=True)
nn.init.normal_(position_embeddings, std=0.02)