Pretraining codeBERT

[hbenyamina]

Was codeBERT trained on all of the 6 programming languages at the same time?
If yes, how? Shuffle everything ? Train on one programming language after the other?
Thank you in advance.

[guody5]

Yes, at the same time. We merge all data and shuffle them.

[hbenyamina]

Great!! thanks for your time.
What about the training objectives, how did you manage to train the model on 2 different datasets? One batch from the NL-PL dataset for MLM then one other batch from the PL dataset for RTD? Or you trained the model on MLM first for some time and then followed by RTD??
I appreciate the time put answering the question.

[guody5]

We first train the model on MLM until convergence, and then we train the model using RTD.

[hbenyamina]

Thank you so much. I appreciate it.

[Meloneneis]

I have a follow-up question. Did you train using RTD until convergence as well and if so, is it correct to say that the longer you train on a second objective, the higher the loss will be on the first objective?