2509 update focalloss to use sigmoid (7/July)

[wyli]

Signed-off-by: Wenqi Li wenqil@nvidia.com

Fixes #2509

Description

using sigmoid activation to align with the literature

the change is verified:
previously gamma=0 reduces it to nn.CrossEntropyLoss
now gamma=0 reduces it to nn.BCEWithLogitsLoss

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

Thanks for the quick fix.
@yiheng-wang-nv could you please help review it?

Thanks.

[yiheng-wang-nv]

Hi @wyli , in paper (https://arxiv.org/pdf/1708.02002.pdf) it uses sigmoid operation to compute p but it also says to use 2-class classification for instance:

However, for multi-class task, I think it still needs to use softmax operation, just as the cross entropy loss does:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#CrossEntropyLoss

Hi @ristoh @ericspod , could you please help to double check it?

[wyli]

https://arxiv.org/pdf/1708.02002.pdf

Hi @yiheng-wang-nv, the footnote you cited is for describing the formulation. In section 4 classification subnet, it's clear: "for each of the A anchors and K object classes... Finally sigmoid activations are attached to output the KA
binary predictions per spatial location"

replacing the sigmoid with softmax should work as well (for classification with mutually exclusive classes), for example in their source code they support both:
https://github.com/facebookresearch/Detectron/blob/1809dd41c1ffc881c0d6b1c16ea38d08894f8b6d/detectron/modeling/retinanet_heads.py#L281-L295

tensorflow implements the sigmoid version https://github.com/tensorflow/addons/blob/e83e71cf07f65773d0f3ba02b6de66ec3b190db7/tensorflow_addons/losses/focal_loss.py

[sandylaker]

However, for multi-class task, I think it still needs to use softmax operation, just as the cross entropy loss does:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#CrossEntropyLoss

Hi @ristoh @ericspod , could you please help to double check it?

I would like to add an extra comment. For anchor-based detector, sigmoid is more widely adopted. So the classification branch actually performs multi-label classification. At inference stage, each anchor box will be classified to the class with highest score, even when the scores do not fulfill sum(softmax(scores)) = 1. Hope it helps.

[ericspod]

I think it's good to go though perhaps some more explanation of the computation would help to clarify why this differs from the "standard" definitions of focal loss most commonly seen, why things are done in log space for instance.