[fix] VisualBERT returns attention tuple #1036

[abhinav-bohra]

PROBLEM: The default value of output_attentions in forward( ) call of BertEncoderJit (in mmf/modules/hf_layers.py) is set as False. So even if the user/developer specifies output_attentions = True in config; its value is taken as default False and thus VisualBERT returns an empty tuple for attentions.

FIX: Set output_attentions as None in BertEncoderJit's forward( ) definition, and update output_attentions to self.output_attentions if it is not passed as an argument (i.e it is None). Therefore, output_attentions will now take the value of self.output_attentions (which was initialized using config during instantiation of BertEncoderJit class)

The issue with output_hidden_states was the same, and it was fixed in a similar way.

Tested locally.

[facebook-github-bot]

Hi @abhinav-bohra!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[apsdehal]

Thanks for making this changes and contributing to MMF. Can you write a simple test to verify that this actually works. Test should go in tests/models/test_visual_bert.py.

[abhinav-bohra]

Thanks for making this changes and contributing to MMF. Can you write a simple test to verify that this actually works. Test should go in tests/models/test_visual_bert.py.

Thanks for your feedback @apsdehal . I will add the tests.

I had one doubt - there can be 3 places where the user can set output_attention, namely;

1. In config
2. During instantiation of the model
3. During the forward pass

For example:

config = BertConfig.from_pretrained('bert-base-uncased', output_attentions = True)
model = VisualBERTBase(config, output_attentions = False)
y, z, attn = model.forward(self.x, output_attentions = False)

Should the priority be: forward > model > config ?
i.e even if output_attentions is set as True in config but passed as False in forward, we should not return attention_list.

Or should it be an 'OR' of all the three cases; i.e. setting it anywhere as True will return the attention_list ?