Finetuning the pretrained model on MSR-VTT

[geyuying]

Hi,

Thanks for your excellent work!

When I finetune the pretrained model that you provide on MSR-VTT, there is a warning shown as below:

"Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertModel: ['vocab_projector.bias', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_layer_norm.weight', 'vocab_projector.weight', 'vocab_layer_norm.bias']"

Is it expected?

Thanks!
Yuying

[m-bain]

I don't seem to encounter this warning, are you using the conda environment given? Whats your huggingface / transformers version?

[bryant1410]

I also see this warning following the setup described in the README.

It should be reproduced with:

from transformers import AutoModel
model = AutoModel.from_pretrained("distilbert-base-uncased")

But I believe it can be safely ignored. It says that some weights were not used by the loaded architecture, so some weights from the checkpoint are ignored. This is probably because it was trained with DistilBertForMaskedLM but it's loading with DistilBertModel. The extra params of the MLM version are vocab_transform, vocab_layer_norm, and vocab_projector, which make sense with the ones printed by the warning. These are used by the MLM training but aren't used in a vanilla model.

[m-bain]

Ok, yeah that's expected - closing this issue :)