Multitask Pre-training

[MichaelFu1998-create]

Dear authors of CodeT5,
Thanks for contributing such an amazing model to the community.
In the paper, it is said that CodeT5 was pre-trained using multiple tasks.
I'm wondering how these task were arranged, did you pre-train multiple tasks all-in-one and combine the loss or did you pre-train each task one by one.
Thank you very much for your help :)

Kind regards
Michael

[yuewang-cuhk]

Hi, we sample one task for each batch based on a balanced sampling strategy during multi-task pretraining. Please refer to here for more details.