New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于预训练target为mlm的问题 #50

Open

cobraheleah opened this issue Jun 9, 2020 · 1 comment

cobraheleah commented Jun 9, 2020

你好，我看代码在构建mlm数据集的过程中，针对一条数据，并没有在开头和结尾加入[CLS]和[SEP]，不知道是代码漏加了还是在训练mlm的时候不需要加入这两个token

Collaborator

zhezhaoa commented Jul 14, 2020

您好，我认为不需要加入
对效果影响很小
下游任务可以把[CLS]训练好

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment