Visualization of BertForSequenceClassification is hard to understand

[davidefiocco]

I am training a BERT trained on the CoLA task (see also #303), so that I can classify sentences as grammatically acceptable or not. Take this example:

"These tests don't work as expected" (grammatically acceptable)
"These tests doesn't work as expected" (unacceptable)

If I run the two examples through the classifier I get as score through softmax in the binary classifier
0.99395967 and 0.00011.

In the two cases, I get interpretations via the notebook in the gist https://gist.github.com/davidefiocco/3e1a0ed030792230a33c726c61f6b3a5 (adapted from the SQuAD example in this repo). And in particular creating the visualization with

score_vis = viz.VisualizationDataRecord(
                        attributions_sum,
                        torch.max(torch.softmax(score[0][0], dim=0)),
                        torch.argmax(score[0][0]),
                        torch.argmax(score[0][0]),
                        text,
                        attributions_sum.sum(),       
                        all_tokens,
                        delta)

print('\033[1m', 'Visualization For Score', '\033[0m')
viz.visualize_text([score_vis])

In the two cases I get

The interpretations look extremely similar in spite of the dramatic change in score, while I would expect the interpretation to change and the model to focus on the grammar mistake (e.g. by focusing on the verb, or the noun)

Am I doing something wrong in my implementation (likely!), or is LayerIntegratedGradients not performing great in this example? Can someone suggest viable alternatives to try out?

[NarineK]

@davidefiocco , from the visualization I can see that attribution label isn't visualized correctly.
I think maybe you are not attributing to the correct output target index. What is the target index that you use ? Is there any ?

score_vis = viz.VisualizationDataRecord(
                        attributions_sum,
                        torch.max(torch.softmax(score[0][0], dim=0)),
                        torch.argmax(score[0][0]), # why is this identical with the next class 
                        torch.argmax(score[0][0]),
                        <THIS SHOULD BE THE TARGET INDEX IF THERE IS NO THEN IT SHOULD BE NONE >,
                        attributions_sum.sum(),       
                        all_tokens,
                        delta)

[NarineK]

Oh, just saw attached notebook, I think that the problem is in the output. I think that you might need to provide a target. You might not have to do:

def custom_forward(inputs):
    out = model(inputs)[0][0]
    return out

but perhaps access the right target from there. My example that I gave you was for my model which is probably very different from yours

[NarineK]

In my case it is this:

def custom_forward(inputs):
    out = model(*inputs)[0]
    return out

You need to know what do you attribute to what. Do you attribute the cat or the dog to the inputs? target is the index of the class - cat index or dog index. In your case you can adjust the classes. See how we use target:
https://captum.ai/tutorials/Multimodal_VQA_Interpret

[davidefiocco]

Thanks @NarineK for your reply!
tl;dr I tried to follow your suggestion and looked also into https://captum.ai/tutorials/IMDB_TorchText_Interpret but I am struggling a bit with the results and I'll leave here some notes to clarify the problem.

---

I have a binary classification problem (CoLA task), and I would like to see in captum which features bring the (softmax) score of class 0 (i.e. ungrammatical sentence) up. Ideally, the interpretation should highlight features that make the sentence not grammatical.

I revised the code and it's visible at https://gist.github.com/davidefiocco/3e1a0ed030792230a33c726c61f6b3a5

The behavior of my model is such that for for some input_ids created from the (ungrammatical) sentence
"These tests does not work as expected."
I get a tuple

> model(input_ids)
(tensor([[ 4.5192, -4.2522]], grad_fn=<AddmmBackward>),)

which contains the logits ("ungrammatical" and "grammatical" respectively) of my binary classifier.

To get the logits only, I created also have a predict method as follows:

def predict(inputs):
    return model(inputs)[0]

so that

> predict(input_ids)
tensor([[ 4.5192, -4.2522]], grad_fn=<AddmmBackward>)

To use the output of the model in captum, I thus have a (modified) custom forward that computes the softmax value for the "ungrammatical" class and now reads:

def custom_forward(inputs):
    preds = predict(inputs)
    return torch.softmax(preds, dim = 1)[0][0].unsqueeze(-1) # second [0] because I care about score of 0th class

so that

> custom_forward(input_ids)
tensor([0.9998], grad_fn=<UnsqueezeBackward0>)

I would like to see which features of my input drive made that score (aka the score of my 0th class) close to 1.

After computing attributions and summarizing them, I invoke viz.VisualizationDataRecord, but I am not sure I am doing this all right:

score = predict(input_ids)  # thus equal to tensor([[ 4.5192, -4.2522]], grad_fn=<AddmmBackward>)

score_vis = viz.VisualizationDataRecord(
                        attributions_sum,
                        torch.softmax(score, dim = 1)[0][0], # probability of my target class
                        torch.argmax(torch.softmax(score, dim = 1)[0]), # the predicted class
                        0, # not sure what I should put here
                        text,
                        attributions_sum.sum(),       
                        all_tokens,
                        delta)

Not sure if someone now sees something which is obviously not OK with all of the above. Full version of the code is at https://gist.github.com/davidefiocco/3e1a0ed030792230a33c726c61f6b3a5

[NarineK]

Hi @davidefiocco , yes, you can do it that way as well. If you only care about the 0(ungrammatical) class. Our visualization functions are just examples. You can also printout the scores and see if the visualization matches the score. The higher the score the more important is the feature for your output. Also, perhaps you want to compute attribution without applying the softmax.
Because of softmax you might get a relatively small range of attribution scores.

[davidefiocco]

I removed the softmax from the custom_forward to make it read

def custom_forward(inputs):
    preds = predict(inputs)
    return preds[0][0].unsqueeze(-1)

The problem still persists though, as my interpretations change little with my input, while the score wildly varies :(

• "These test fails."
  custom_forward(input_ids) = tensor(-3.5023], grad_fn=<UnsqueezeBackward0>)
  

• "These tests fail"
  custom_forward(input_ids) = tensor([4.2576], grad_fn=<UnsqueezeBackward0>)
  

[NarineK]

@davidefiocco , the attribution could be correct but the visualization is tricky because I can see that the first example has much larger negative attribution. The word fails is positive but it is probably very small positive value. Attribution is not perfect and it looks like it is a bit confused with word fails When we normalize attributions then the color coding also gets adjusted. You can try to normalize the attribution across those 2 samples. Also looking into the distribution of attributions for each embedding vector can be helpful.

The normalization happens here:
attributions = attributions / torch.norm(attributions)
I'd try without normalization or try a different normalization technique.

[NarineK]

@davidefiocco, have you made any progress here ?

[davidefiocco]

Hi @NarineK thanks a lot for following up here! I had tried eliminating normalization, but that didn't solve it. I'll try to give it another shot and will update the issue here with my findings.

[davidefiocco]

Hi @NarineK , I hope you're well, apologies for the delay.

I could not solve the problem, but I am more convinced that there is some bug in my code now.

The good news is that now my code is perfectly reproducible at least, as for debugging I borrowed a (binary) sentiment classifier whose weights are publicly downloadable via the HuggingFace great collection at https://huggingface.co/lvwerra/bert-imdb (trained on IMDB)

Sentiment can be reversed simply by changing one word, and so it should be easier to debug.

The notebook for now is runnable without modifications by making a copy of what you find at
https://colab.research.google.com/drive/1snFbxdVDtL3JEFW7GNfRs1PZKgNHfoNz
If one doesn't have/want to use a Google account to run colab, the notebook should be executable locally on a linux machine having wget and pip installed.

The resulting visualization is

which seems buggy to me. I'll try to look at the code more closely, but if you have hints that would be totally welcome!

P.S.: I put the same code on https://gist.github.com/davidefiocco/40a1395e895174a4e4d3ed424a5d388a also, for reference.

[NarineK]

@davidefiocco , try to change the text a little bit and see how your attribution changes. This is actually a very interesting example and it shows that the network is prone to adversarial attacks and is possibly easy to fool.
Try to remove the dot at the end of the sentence and see how the attribution changes.
Now, try something like this:
text = "The movie was one of those amazing movies you can not forget"
or this:
text = "The movie was one of those amazing movies"
The attribution starts to make more sense ?

[NarineK]

Try also to increase n_steps and observe the delta:

attributions, delta = lig.attribute(inputs=input_ids,
                                    baselines=ref_input_ids,
                                    n_steps=700,
                                    internal_batch_size=3,
                                    return_convergence_delta=True)
delta

[NarineK]

from what I see you are attributing to the negative class. Positive (good recommendation) class would be ?

def custom_forward(inputs):
    preds = predict(inputs)
    return torch.softmax(preds, dim = 1)[0][1].unsqueeze(-1)

[davidefiocco]

Hi @NarineK, thanks!

I tried to change only the input (by shortening the sentence) and attribute the negative label (so no other change in the notebook, just the input). I get:

And this is what I get with a similar, but negative "micro-review":

I computed attributions and convergence deltas for various values of n_steps, and by eyeballing the results convergence seems reached at ~200:

Increasing the delta beyond the default value of 50 doesn't dramatically change the interpretation I visualize though.

Also as for your question, indeed the custom forward returning the softmax score for the positive class would be written as

def custom_forward(inputs):
    preds = predict(inputs)
    return torch.softmax(preds, dim = 1)[0][1].unsqueeze(-1)

But also playing with that definition doesn't yield results that make sense intuitively to me (I would like the interpretation to be focused on the adjective, which sets the sentiment of the review).