Get embeddings of language encoder and Object-Relationship Encoder?

[yezhengli-Mr9]

Any suggestion of getting embeddings of language encoder and Object-Relationship Encoder? I refers to output_hidden_state mentioned in this blog as well as the official tutorial.

[zhmd]

I'm not 100% sure, but if you refer to the encoded representation before the cross-modality interaction module, then probably you need to hack these two lines and save these intermediate representations:

lxmert/src/lxrt/modeling.py

Lines 554 to 559 in 0db1182

	for layer_module in self.layer:
	lang_feats = layer_module(lang_feats, lang_attention_mask)
	# Run relational layers
	for layer_module in self.r_layers:
	visn_feats = layer_module(visn_feats, visn_attention_mask)

If you need the ones after the cross-modality interaction module, then just take these two:

lxmert/src/lxrt/modeling.py

Line 566 in 0db1182

return lang_feats, visn_feats

[yezhengli-Mr9]

I'm not 100% sure, but if you refer to the encoded representation before the cross-modality interaction module, then probably you need to hack these two lines and save these intermediate representations:

lxmert/src/lxrt/modeling.py

Lines 554 to 559 in 0db1182

	for layer_module in self.layer:
	lang_feats = layer_module(lang_feats, lang_attention_mask)
	# Run relational layers
	for layer_module in self.r_layers:
	visn_feats = layer_module(visn_feats, visn_attention_mask)

If you need the ones after the cross-modality interaction module, then just take these two:

lxmert/src/lxrt/modeling.py

Line 566 in 0db1182

return lang_feats, visn_feats

Sort of help, thanks! @zhmd