use Multilingual pretrain model Bert

[kewin1807]

please tell me, can i use Multilingual pretrain model from Bert to train custom data with albert code ???

[brightmart]

you can have a try.
and be aware that there are some differences between bert and albert in modelling.py
why do you want to train multillingual model?

[kewin1807]

i want to use model with vietnamese language. The important of change is share parameters, i know. how i can train with my language. Thanks for support =)

[brightmart]

1、you can change vocab.txt in ./albert_config, then set non_chinese to True when create pretrain data using create_pretraining_data.py
2、then do pre train using run_pretraining.py

[kewin1807]

okay. Thanks for support. Best repo =)

[kewin1807]

I have tried to pretrain with my dataset, but i see the loss is very small but accuracy is not improve. How i can improve result

[geekboood]

@brightmart Can we have a multilingual model for just Chinese and English? Cause in practical scenerios we may meet many english words in APP names, music names, Apple's all products's name and so on, and Google's multilingual model has too many languages. Our daliy life cannot leave English, you can see that Apple try to use purely Chinese in its products, such as replace Finder with 访达 which I think is totally a mess.
Maybe a language model for just Chinese and English can have huge impact on both research and industry and many multilingual tasks can benefit from it.