Loading fine tuned BertForMaskedLM

[tgriseau]

Hi,

I tried to fine tune BertForMaskedLM and it works. But i'm facing issues when I try to load the fine tuned model.
Here is the code I used to load the model :

model_state_dict = torch.load("./finetunedmodel/pytorch_model.bin",  map_location='cpu')
model_fine = BertForMaskedLM.from_pretrained(pretrained_model_name='bert-base-multilingual-cased', state_dict=model_state_dict, cache_dir='./data')

The error I'm facing is : init() got an unexpected keyword argument 'state_dict'

Does someone already faced this issue ?

Thanks

Edit : I trained my model on gpu and try to use it on cpu. When I use it on gpu it works !

[thomwolf]

Maybe update to a recent version of pytorch-pretrained-bert?

[tgriseau]

I'm already using the last release.

I don't have any issues running it on gpu. The problem append when using map_location

[MuruganR96]

yeah yes. if you trained model in GPU, can't be loaded. we will change map_location="CPU" in modeling.py line 511.
https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/pytorch_pretrained_bert/modeling.py#L511

state_dict = torch.load(weights_path, map_location="CPU" )

now it will load our finetuned model. this is only for inference running on CPU.

[tgriseau]

All good ! It works, thanks.

Maybe adding a device parameter to the function from_pretrained could be usefull.

Thanks for your help.

[thomwolf]

Great, thanks @MuruganR96

[psureshmagadi17]

@tgriseau - I want to fine-tune bert on MaskedLM using domain-specific text. could you please provide an example of how you fine-tuned or provide some details about what kind of inputs need to be passed? will I be using the true sentence as the output for fine-tuning?