Optional layers

[jplu]

What does this PR do?

This PR adds the possibility to have optional layers in the models thanks to the new input/output process. Here the pooling layer is created or not for the BERT/ALBERT/Longformer/MobileBERT/Roberta models. The keys to ignore when loading for these layers has been updated in same time.

[LysandreJik]

Thanks for working on this @jplu. I think we should take the opportunity to think about this issue: #8793.

The problem with the add_pooling_layer option and how it's currently done in PyTorch models is that when doing a training initialized from a model checkpoints that contains the pooling layer, like bert-base-cased:

model = BertForMaskedLM.from_pretrained("bert-base-cased")
# Fine-tune the model on an MLM task

we're losing the pooling layer doing so. It's not a big deal here as we're doing an MLM task, however, if we want to use that model for a downstream task:

model.save_pretrained("bert-base-cased-finetuned-mlm")
classifier_model = BertForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mlm")

we're now having a classifier model that has a randomly initialized pooling layer, whereas the weights that were stored in the bert-base-cased original checkpoint would have been better than a randomly initialized layer.

The issue is that right now, we have no way of specifying if we want to keep the pooling layer or not in such a setup. I would argue that controlling it from the configuration would really be useful here, rather than setting it to add_pooling_layer=False in architectures that do not need it.

cc @jplu @sgugger @patrickvonplaten

[jplu]

Indeed, it starts to be more complicated than we thought at the beginning, but the case you are raising is a very good one!!

I think that controlling this from the config to have the same behavior would be more flexible, I +1 this proposal!

[patrickvonplaten]

The logits bias can be fine-tuned, so not sure that we want to have those in _keys_to_ignore_on_load_missing. Or are they missing from all bart models (and thus set to 0?)

[jplu]

That's a good question, I don't know enough BART to answer to the question, so I will remove that one 👍

[patrickvonplaten]

Very much like this PR!

[patrickvonplaten]

Thanks for working on this @jplu. I think we should take the opportunity to think about this issue: #8793.

The problem with the add_pooling_layer option and how it's currently done in PyTorch models is that when doing a training initialized from a model checkpoints that contains the pooling layer, like bert-base-cased:

model = BertForMaskedLM.from_pretrained("bert-base-cased")
# Fine-tune the model on an MLM task

we're losing the pooling layer doing so. It's not a big deal here as we're doing an MLM task, however, if we want to use that model for a downstream task:

model.save_pretrained("bert-base-cased-finetuned-mlm")
classifier_model = BertForSequenceClassification.from_pretrained("bert-base-cased-finetuned-mlm")

we're now having a classifier model that has a randomly initialized pooling layer, whereas the weights that were stored in the bert-base-cased original checkpoint would have been better than a randomly initialized layer.

The issue is that right now, we have no way of specifying if we want to keep the pooling layer or not in such a setup. I would argue that controlling it from the configuration would really be useful here, rather than setting it to add_pooling_layer=False in architectures that do not need it.

cc @jplu @sgugger @patrickvonplaten

I remember that we were thinking about adding a config param for add_pooling_layer for PT: #7272 and decided not to. I still think the cleaner solution is to not add a config param because it's a very weird use-case IMO. Why wouldn't the user just use a BertForPreTraining model for his use case? But I'm also fine with adding a config param instead. It's not a big deal to me...but in this case I'd definitely prefer to not add it to the general PretrainedConfig, but to each model's config.

[LysandreJik]

Good point regarding the BertForPreTraining. I think this is a use-case (you want to keep a layer from another architecture) where you would want to build your own architectures for that, to have complete control over the layers.

I think we might be missing some documentation on how to do that, and on how creating an architecture that inherits from PreTrainedModel works, but this is a discussion for another time.

Ok to keep it this way.

[sgugger]

It all looks good to me, thanks for working on this!

[LysandreJik]

LGTM, great work @jplu!

[LysandreJik]

Was this previously missing? It is in the TFPretrainedBartModel:

class TFPretrainedBartModel(TFPreTrainedModel):
    config_class = BartConfig
    base_model_prefix = "model"

[jplu]

LGTM for me!