Suggestions for implement pre-training

[WeihongM]

Thanks for your impressive work.
Can you share how to implement pretraining code?

[tghong]

Thank you for your interest in our work!
We have no plans to provide a pre-training code yet.
You can implement your own pre-training script by referring the Hugging Face's transformers code: https://github.com/huggingface/transformers/blob/v4.19.2/src/transformers/data/data_collator.py#L748