Why vocabulary is divided by GPU number and how to load it?

[Aurora-slz]

When I train a pretraining model, the vocabulary is divided by the number of GPU. So I can't directly load it with origin model in downstream tasks.
How should I do?Thanks!

[duzx16]

The vocabulary size is divided by MP_SIZE (which is the model parallel size). Did you set MP_SIZE to the number of GPUs? To use the original model, MP_SIZE should be set to 1.

[SeonggwanAhn]

In addition, you can refer to this for setting MP_SIZE.

[Aurora-slz]

Thanks! I used the change_mp.py to combine them into one and solve the problem.