Lost gradient after clipping in focal loss?

[danijar]

addons/tensorflow_addons/losses/focal_loss.py

Line 132 in 776b751

y_pred = K.clip(y_pred, K.epsilon(), 1. - K.epsilon())

[seanpmorgan]

cc @AakashKumarNain code owner

[AakashKumarNain]

@danijar any sample code to reproduce?

If it is the case though, then I suspect that epsilon added to y_pred in this line

p_t = (y_true * y_pred) + ((1 - y_true) * (1 - y_pred))

should solve the issue. Need to test it though.

[danijar]

I just read through the code and wanted to highlight a potential problem. Looking at K.clip, it doesn't seem to provide a gradient for values that have been clipped. In this is indeed the case, a fix would be

clipped = K.clip(y_pred, K.epsilon(), 1. - K.epsilon())
y_pred = y_pred - tf.stop_gradient(y_pred) + tf.stop_gradient(clipped)

[AakashKumarNain]

Thank you @danijar for looking into this and providing a fix. I will look into it once again and we will make the necessary code changes. Thank you again

@WindQAQ @facaiy @seanpmorgan sounds good?

[WindQAQ]

@AakashKumarNain Sorry for the late response. I'm fine with this. Maybe provide a testcase to verify the patch is needed!

[AakashKumarNain]

Sorry for the delay. I was on vacation. I will check and if needed, fix this during this weekend

[AakashKumarNain]

K.clip is just clip by value. It doesn't seem to affect the gradients, hence no fix is needed IMO. Though looking for any additional inputs

[danijar]

What do you mean? Clip by value is not differentiable.

[AakashKumarNain]

The same thing is being used here in binary_crossentropy calculations.
https://github.com/tensorflow/tensorflow/blob/r2.0/tensorflow/python/keras/backend.py#L4558-L4588

[Squadrick]

@danijar I understand the confusion.

tf.clip_by_value uses tf.minimum and tf.maximum currently (ignoring the TODO).

Both tf.minimum and tf.maximum have their gradients defined here. So yes, as @AakashKumarNain mentioned, using tf.clip_by_value shouldn't be a problem for differentiation.

As for the gradient, it'll be 0 outside the range of min_value and max_value.

[danijar]

Exactly. Is it supposed to be 0 outside of the range?

[Squadrick]

@danijar My bad, you're right. A clip is used in BCE before taking a log for numerical stability. That won't be needed for focal loss.

Funnily enough in the tf.keras implementation of BCE, that does both a clip followed by an output + eps. I think the addition of eps after the clip won't be needed.

[danijar]

My main point was that the gradient should probably be patched back onto the clipped tensor. Anyway, I'm not a user of the module.

[AakashKumarNain]

We will patch it.