I am also very much interested in SHAP with LLMs.

We can't seem to get the explainer to even execute.

@codybum probably a different issue than this bug report, but I ran into something similar. Do you have return_all_values=True in your pipeline or top_k=None?

@derekelewis thanks for the response!

I am doing text generation not classification, so that could be part of my issue. I tried to follow the SHAP docs for text generation, but there was nothing specific to llama.

I had previously attempted to set "return_all_value=True" based on your code, but it could cause the following error: "ValueError: The following model_kwargs are not used by the model: ['return_all_scores'] (note: typos in the generate arguments will also show up in this list)"

import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import pipeline
import numpy as np
import shap
import torch
from transformers import pipeline
import shap


tokenizer = AutoTokenizer.from_pretrained("models/llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("models/llama-2-7b-hf")


pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.float16,
    trust_remote_code=True,
    #return_all_scores=True,
    top_k=None,
    device_map="auto",
)


explainer = shap.Explainer(pipeline)
shap_values = explainer(["hello, world!"])

print(shap_values)

Does anyone have a working transformer example?

After some debugging, I modified the text-generation example of GPT-2 a bit and got it to work with GPT-J.

import numpy as np
from transformers import AutoTokenizer, AutoModelForCausalLM
import shap
import torch

tokenizer = AutoTokenizer.from_pretrained('nlpcloud/instruct-gpt-j-fp16', use_fast=True)
model = AutoModelForCausalLM.from_pretrained('nlpcloud/instruct-gpt-j-fp16', torch_dtype=torch.float16).cuda()

model.config.is_decoder=True

gen_dict = dict(
    max_new_tokens=10, 
    num_beams=5,
    renormalize_logits=True,
    no_repeat_ngram_size=8, 
)
model.config.task_specific_params = dict()
model.config.task_specific_params["text-generation"] = gen_dict

shap_model = shap.models.TeacherForcing(model, tokenizer)
masker = shap.maskers.Text(tokenizer, mask_token="...", collapse_mask_token=True)
explainer = shap.Explainer(shap_model, tokenizer)
shap_values = explainer([prompt])

Yet, when I swap the above with OpenLlama2, I got weird results.
As the image shows 2 things:

1. Unlike GPT-J where the arguments for generation (i.e. num_beams, renormalize_logits) had been used successfully, the shap explainer did not applied them to OpenLlama2 (actually deleting some of them, such as "num_beams", when I checked the model's config before and after running shap's explanation).
2. Instead of calculating shap score of outputs conditioned on the inputs, as far as I can observe, the shap scores of the OpenLlama outputs are calculated conditioned on the outputs themselves...

I was wondering if there has been any progress in this area? Is the computational cost too high for this to be practical on large models?

I am also curious to see if anyone has successfully written a code with recent models such as medalpaca or llama-series models. If yes, can you share the code to ybkim95@mit.edu please?

I am also very much interested in SHAP with LLMs.

We can't seem to get the explainer to even execute.

Your Issues should be connected to shap not handling llama as a huggingface transformers model.
It's easily fixable, by updating the transformers file in shap. Then the Explainer should run successfully (to a certain point).

I got it to work with Mistral, haven't come around to do it with LlaMa 2 yet. But I had the same error and would think the fix is the same too.

However afterwards you will run in another error/weird plots (as mentioned by @LittleDijkstraZ ).

I was wondering if there has been any progress in this area. Is the computational cost too high for this to be practical on large models?

I'm not sure if there is a fundamental reason why it shouldn't work based on the size.

The computational cost certainly is high though and it runs a long time. My Partition Explainer with Mistral runs about 4 Minutes on 50 new tokens with very small input. And the values of course must be wrong. Currently, this is my output:

I think it's a problem of implementation. I'd be willing to do a PR on this and the other changes plus update the example notebook... but I'm not sure where to start. Maybe one of the maintainers can help? (@connortann ?) - I'm not sure how this all works so I'll just tag you here, sorry if that's annoying/not the correct approach.

PRs to investigate this issue and get this fixed would be most welcome! I think it's been quite a while since the text explanation functionality has had significant active development, but hopefully together we can figure out a fix. It would be great to get this sorted.

There some existing tests that use a tiny test model here:

A good place to start would probably be to get a small fast minimal reproducible example - ideally with a synthetic dataset and a dummy tiny language model. Hopefully we can pin down the root issue without using a full llama-2

Thank you for the reply @connortann ! I will open a PR once I am closer to the solution.

It's not a general transformer error. I'm successfully running it with GPT-2, Dialog GPT, and GODEL - the results look good to me.

Therefore in my mind it must be an issue with the models and/or masking and not the calculation of SHAP values. As the Explainer is handling a model wrapped by the teacher forcing class in all these cases. This would lead me to think that it's a problem with the transformer's implementation of those particular models (mistral/llama2) or the model architecture.

I'll update this with some more details and logs when I have the time to try it out.

Hello again. I have had some more time for debugging and was able to drill down the issue.
The calculated log odds (by the teacher forcing model) are wrong. Or to be exact, they don't change from the baseline and the explainer is stuck on them. That's why the SHAP values are 0 for all tokens in the end.

I think this is because of the full_masking call. Running log odds calculation separately with the teacher forcing model seems fine to me. Any insights on this full masking call would be appreciated.

GPT-2 Example (how it should be)

Mistral AI

FYI, Captum now has included KernelSHAP for explaining LLMs. Read more here: https://captum.ai/tutorials/Llama2_LLM_Attribution

Yea @amir-rahnama , this is what I used for my thesis project in the end, the performance compared to PartitionSHAP is not great though. Using a GPU you can achieve good speed but it's quite resource consuming.

However it seems the underlying issue is already fixed, which is great @costrau !