Deep SHAP giving weird outliers

[Janerlend99]

Hi guys,

I've been using Deep SHAP in the SUMO-RL traffic environment, and I have encountered some unusual results.

This is the model wrapper I use on the PPO policy that is input to shap.DeepExplainer

    class sb3Wrapper(nn.Module):
        def __init__(self, model):
            super(sb3Wrapper, self).__init__()
            self.extractor = model.policy.mlp_extractor
            self.policy_net = model.policy.mlp_extractor.policy_net
            self.action_net = model.policy.action_net

            self.soft_max = torch.nn.Softmax(dim=-1)

        def forward(self, x):
            x = self.policy_net(x)
            x = self.action_net(x)

            # Transform log probabilities to normal probabilities

            x = self.soft_max(x)

            return x

These are the results I get:

Some instances look reasonable, while some instances look like this:

This is the results I get when not adding the soft_max layer or if I use KernelSHAP instead:

Does anyone know the reason for why I get such large shap values for some features in a single instance when using soft max?

Also, in the start I used this wrapper but found out that DeepExplainer did not support the operations with x in the forward function. I mean, the code ran successfully but the results was definately not correct. Can anyone explain why I cant define the forward function in this way:

    class sb3Wrapper(nn.Module):
        def __init__(self, model):
            super(sb3Wrapper,self).__init__()
            self.extractor = model.policy.mlp_extractor
            self.policy_net = model.policy.mlp_extractor.policy_net
            self.action_net = model.policy.action_net

        def forward(self,x):
            x = self.policy_net(x)
            x = self.action_net(x)

            # Transform log probabilities to normal probabilities
            x = torch.exp(x)
            
            # Normalize the probabilities
            x = x / x.sum(dim=-1, keepdim=True)  # Assuming probabilities are along the last dimension
            
            return x