对xlnet预训练过程的一点疑问

[genggui001]

对于一段文本，选取其中的K个单词，每次只MASK掉一个，生成K条训练数据，再最大化K条训练数据的对应正确单词的对数概率。

是不是也可以达到和xlnet一样的效果？

[zihangdai]

Objective-wise, this is the same as the XLNet. However, this procedure requires K different forward and backward passes, which makes it too expensive to use,

[kimiyoung]

@genggui001 That would take 85x more machines, which is almost impossible to train. Also, given 85x more machines, simply scaling up XLNet will probably be better due to better data efficiency.

[guotong1988]

Thank you. Learned a lot here.