Are codes for pretraining available?

[PrettyMeng]

It seems that this repository only contains the code to perform finetuning pretrained RoBERTa. Are code for pretraining available now? Can you possibly add some command example for doing TAPT? Any advice or explanation will be highly appreciated. Thanks in advance!

[kernelmachine]

Hi there, check the ADAPTIVE_PRETRAINING.mdfile for DAPT/TAPT commands

[PrettyMeng]

Hi there, check the ADAPTIVE_PRETRAINING.mdfile for DAPT/TAPT commands

Thanks for your quick reply! But just to clarify, in my case, the script is not working when transformers==2.4.1 as in environment.yml. In my case, it works well when transformers==2.8.0.

[shizhediao]

Hi there, check the ADAPTIVE_PRETRAINING.mdfile for DAPT/TAPT commands

Thanks for your quick reply! But just to clarify, in my case, the script is not working when transformers==2.4.1 as in environment.yml. In my case, it works well when transformers==2.8.0.

Yes, same here.
When I am trying to run DAPT, there is an error ""ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING""
I need to upgrade the transformers to 2.8.0 and it works well.
@kernelmachine