How do I use this tool for my own model?

[akshaysadanand]

Hi, I have trained an XLM model that translates from English to Spanish. A model for this language pair is not available on huggingface's repo. Is there any way to load my saved model?

[jessevig]

Hi, I assume you are still using the Huggingface XLMModel class, but just with your own saved model weights? In that case you can still use the code in https://github.com/jessevig/bertviz/blob/master/head_view_xlm.ipynb but just change following lines, so that model_version is the directory in which your model is saved.

model_version = 'xlm-mlm-ende-1024'
model = XLMModel.from_pretrained(model_version, output_attentions=True)

Just be sure to set output_attentions=True, as above. Does that answer your question?

[akshaysadanand]

No, I am not using huggingface's model. I am running the model found on the facebookresearch repo.

[jessevig]

If that is the case, and you are using the head view or model view, you can still use the notebook above, but you would need to call your model instead of the huggingface one and somehow retrieve the attention weights and reformat them as specified in head_view.py:

def head_view(attention, tokens, sentence_b_start = None, prettify_tokens=True):
    """Render head view
        Args:
            attention: list of ``torch.FloatTensor``(one for each layer) of shape
                ``(batch_size(must be 1), num_heads, sequence_length, sequence_length)``
            tokens: list of tokens
            sentence_b_index: index of first wordpiece in sentence B if input text is sentence pair (optional)
            prettify_tokens: indicates whether to remove special characters in wordpieces, e.g. Ġ
    """

I haven't worked with the FB model so not sure how to implement that on the model end.

[akshaysadanand]

Okay, I'll try that. Thanks for the help!

[jessevig]

The other option is to try to convert your model to a huggingface-compatible model but I'm not sure how easy that would be either. It seems like something others might have done though.

[akshaysadanand]

I will look into that. Thank you!