Post-paper Detection Optimizations

[datumbox]

Related to #5307 and #2263. Touches upon requests recorded at #4932 and #5325.

Our target is to improve the existing RetinaNet, FasterRCNN and MaskRCNN architectures using post-paper optimizations:

• Adds a new RetinaNet variant with post-paper optimizations proposed at "Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection".
• Adds a new FasterRCNN and MaskRCNN variants with post-paper optimizations proposed at "Benchmarking Detection Transfer Learning with Vision Transformers" by @rbgirshick and @pdollar
• Adds support of elements from @vaibhava0's recipe, @xiaohu2015's recipe and @fmassa's work on DETR. Other recipes influencing this work is the ones of Swin Transformers and XCiT.

This PR contains commits which were later split on separate PRs to assist the reviews. Currently it contains the following changes on top of the main branch:

• Adds support of norm_layer in all Detection heads.
• Adds a new extendible FastRCNNConvFCHead which follows a similar strategy as the existing MaskRCNNHeads.
• Replaces Conv-Norm-BNs with Conv2dNormActivation where possible; we use PyTorch core's _load_from_state_dict() approach to maintain BC.
• Extends RPNHead to support heavier heads.
• Adds an experimental private _box_loss utility which allows for training models with different box losses. No plans to make this public; the entire API is kept private so that we can review our Detection API as a whole and examine the best way to support different Transforms and Losses in Detection models.
• Adds new 3 variants for RetinaNet and *RCNN based on follow up papers.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 6488c41 (more details on the Dr. CI page):

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[fmassa]

LGTM!

I've left just a minor comment regarding naming, but it's just for discussion and there is no need to act on it

[d4l3k]

@datumbox I'm getting errors when trying to load a model using FPN as a frozen layer.

Traceback (most recent call last):
  File "/mnt/ext/openape/apedepth/train.py", line 85, in <module>
    model.load_state_dict(state_dict, strict=False)
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1566, in load_state_dict
    load(self)
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1564, in load
    load(child, prefix + name + '.')
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1564, in load
    load(child, prefix + name + '.')
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1564, in load
    load(child, prefix + name + '.')
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1560, in load
    module._load_from_state_dict(
  File "/home/rice/venvs/openape/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 131, in _load_from_state_dict
    state_dict[new_key] = state_dict.pop(old_key)
KeyError: 'semantic.backbone.fpn.inner_blocks.0.weight'

This seems to be a regression since PT 1.11 stable and I can no longer save/load my model. Excluding the semantic weights when calling torch.load doesn't help either.

My model has

        self.semantic = freeze(models.detection.fasterrcnn_mobilenet_v3_large_fpn(pretrained=True))

[d4l3k]

Adding a if old_key in state_dict in a few spots seems to fix it.

        version = local_metadata.get("version", None)

        if version is None or version < 2:
            for type in ["weight", "bias"]:
                old_key = f"{prefix}conv.{type}"
                new_key = f"{prefix}conv.0.0.{type}"
+                if old_key in state_dict:
                    state_dict[new_key] = state_dict.pop(old_key)

[datumbox]

@d4l3k Thanks for the heads up.

This seems to be a regression since PT 1.11 stable and I can no longer save/load my model.

If this is true, it's probably not related to this PR. To reduce any confusion, could you please open a separate issue and provide a way to reproduce the problem?

Adding a if old_key in state_dict in a few spots seems to fix it.

This might indicate that the old structure is modified and not compatible with v1. The proposed patch is not a solution but rather more likely to mask the issue. If you provide a way to reproduce the issue, we can help you invstigate.