You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi~, Thanks for such great work! I saw you released the baseline performance of the vanilla KD on the one-stage detector RetinaNet, I wonder how this method is applied. Since the classification prediction of RetinaNet is activated by sigmoid and formulated as multiple binary classification problems solved with Focal Loss, it seems we can not use the vanilla KD on these classification outputs. The output processed by sigmoid, for example: [0.4, 0.7, 0.3, 0.2], is not sum up to 1, obviously. So, how the vanilla KD with KLDiv loss is applied under such a situation? Thanks.
The text was updated successfully, but these errors were encountered:
Thanks for your attention of our work. The distillation on sigmoid probabilistic distributions can also be conducted using KL divergence loss (see Kullback–Leibler divergence, it does not require the sum of the vector to be 1). We use Equation (1) as the form of sigmoid KD loss in our RetinaNet experiments and only replace the softmax function to sigmoid.
BTW, for detectors using sigmoid focal loss, it is more practical to use the above sigmoid KL divergence loss with focal weight.
Hi~, Thanks for such great work! I saw you released the baseline performance of the vanilla KD on the one-stage detector RetinaNet, I wonder how this method is applied. Since the classification prediction of RetinaNet is activated by sigmoid and formulated as multiple binary classification problems solved with Focal Loss, it seems we can not use the vanilla KD on these classification outputs. The output processed by sigmoid, for example: [0.4, 0.7, 0.3, 0.2], is not sum up to 1, obviously. So, how the vanilla KD with KLDiv loss is applied under such a situation? Thanks.
The text was updated successfully, but these errors were encountered: