LRP: AlphaBeta and Zplus

[sebastian-lapuschkin]

I have found a bug in the code for the AlphaBeta rules and also the Zplus rule.

The current code creates copies of the layer's weight and thesholds it at zero into positive and negative weight arrays.
However, AlphaBeta and also Zplus require forward transformations thresholded at zero, which, with the current implementation, requires layer inputs x to be greater than or equal to zero.

For a correct implementation of both rules (Zplus can inherit from AlphaBeta and realize alpha=1, beta=0), it would suffice to copy the layer's weights once, but then compute two forward passes which need to be thresholded.
Does this interfere with the computation of the backward gradient?

Also, for correctly considering the positive and negative forward contributions originating from the bias, the bias needs to be thresholded as well.

Please have a look at (this)[https://github.com/sebastian-lapuschkin/lrp_toolbox/blob/python-wip/python/modules/linear.py], lines 219+, which implements the rules using numpy.
Can this (conveniently) be fixed, without negatively interfering with the gradient computations?

[albermax]

I agree, well detected, I think there is a todo for that?
It should work for relus but not other activation functions.

The biases are part of the get_weights. Therefore the list-comprehension.

Yes, this should quite easily be fixable. I will come back to this, I don't manage today to give you complete response. I follow up!

[albermax]

Ok the todo is cryptic :-)

Yes, this is definitely possible to implement with the gradients.

So the point would be that negative inputs times negative weights should be added to the alpha component and negative inputs times positive weights to the beta component. Right?

        # Distinguish postive and negative inputs.                                                                                                                                   
        Xs_pos = kutils.apply(keras.layers.ReLU, Xs)
        Xs_neg = kutils.apply(keras.layers.ReLU,
                                kutils.apply(times_minus_one, Xs))
	# Compute positive and negative relevance.                                                                                                                                   
        add = keras.layers.Add()
        positive_part = add(f(self._layer_wo_act_positive, Xs_pos),
                            f(self._layer_wo_act_negative, Xs_neg))
        negative_part = add(f(self._layer_wo_act_negative, Xs_pos),
                            f(self._layer_wo_act_positive, Xs_neg))

times_minus_one can be implemented like times_alpha.

What do you think about this?

If you have a computationally faster implementation, that would also be great!

[sebastian-lapuschkin]

looks good. I will try to add this if I find no corresponding commit from you

[sebastian-lapuschkin]

There has been another bug with Xs_neg, which has been fixed.

ZPlus does now inherit from AlphaBeta, sets alpha=1, beta=0 and ignores the bias.
I have kept the existing implementation of Zplus as ZPlusFast, since 1) it is faster due to 2) only computing what needs to be computed assuming x>=0 and w>=0 holds. Thus, expect different results for ZPlus and ZPlusFast for layers which do not ensure both conditions for all layers.

A second pair of eyes going over the code would be nice.
Also: How do I add assertions to the computational graph, which at the least spit out warnings if some conditions do not hold?

[sebastian-lapuschkin]

fyi relevance maps as they are now:

imagenet_lrp_vgg16.pdf

note the difference of ZPlus (which inherits from AlphaBeta) and ZPlusFast. Handling the input layer explicitly for both methods should yield the same results.

[albermax]

I will look into the code and let you know!

I think the easiest is if you write an issue and let me know which assertions you want to have and I will add them.

[sebastian-lapuschkin]

assertions which come to mind right now, wrt ZPlus are

1. layer inputs >= 0
   1.1) in general
2. generic comparisions of two tensors as equal, or at least almost_equal (within relative tolerance)

(2) would greatly help for debugging and optimizing code.

[albermax]

Please let's discuss this best online.

[sebastian-lapuschkin]

I have just updated the AlphaBeta rule class in relevance_based.py, since the previous version was incorrect.
I will for now keep the old implementation as apply_buggy(...) for reference, until functionality has been confirmed.
The code in apply_buggy(...) performs relevance attribution on only half of the positive and negative relevances each, of which the results are then aggregated, which breaks the denominator.

The new implementation was intended to fix that. Could you please verify it does what it should?
see the attached PDF for an explanation of what it should do.
on_alpha_beta.pdf

As a drawback, the current implementation of apply(...) is computationally more expensive.
Can we optimize that?