Does fine-tuned model use the same vocabulary as pre-trained model?

[zhangguanqun]

The pre-trained model use learned joint vocabulary with 64867 tokens, merged by 32k operations. That is the file vocab.bpe.32000.
Does fine-tuned model (e.g. en2de) released in icloud use the same vocabulary file as pre-trained model?

[linzehui]

yes

[zhangguanqun]

appreciate

[zhangguanqun]

If I want to fine-tune this model to support new languages, new tokens should be added to existing file?
That is, new file has larger vocabulary size than 64867?
If do so, the embedding params of checkpoint (e.g. pretrain_checkpoint_last_RAS.pt) released in this project should be expanded before fine-tuning.

[PANXiao1994]

If I want to fine-tune this model to support new languages, new tokens should be added to existing file?
That is, new file has larger vocabulary size than 64867?
If do so, the embedding params of checkpoint (e.g. pretrain_checkpoint_last_RAS.pt) released in this project should be expanded before fine-tuning.

Yes, you are right, if you want to expand the supported languages, you need to merge the newly added tokens into the existing vocabulary, then randomly initialize the embedding vectors of the new tokens.

[zhangguanqun]

appreciate * 2