COCO fine-tuning without panoptic stuffthingmaps

[titusss]

Hello thank you for the great work,

I want to fine-tune the pre-trained CBnet on a COCO-like custom dataset. However, the htc_cbv2_swin_base_adamw_20e_coco.py config expects COCO-stuff pixelmaps. (See configs/BigDetection/cbnetv2/htc_cbv2_swin_base_adamw_20e_coco.py)

In line 92 it tries to load stuffthingmaps into the data dict
seg_prefix=data_root + 'stuffthingmaps/train2017/',

The whole code block, starting at line 89:

samples_per_gpu=1
data = dict(samples_per_gpu=samples_per_gpu,
            train=dict(
                seg_prefix=data_root + 'stuffthingmaps/train2017/',
                pipeline=train_pipeline),
            val=dict(pipeline=test_pipeline),
            test=dict(pipeline=test_pipeline))
optimizer = dict(_delete_=True, type='AdamW', lr=0.0001*(samples_per_gpu/2), betas=(0.9, 0.999), weight_decay=0.05,
                 paramwise_cfg=dict(custom_keys={'absolute_pos_embed': dict(decay_mult=0.),
                                                 'relative_position_bias_table': dict(decay_mult=0.),
                                                 'norm': dict(decay_mult=0.)}))

What if I do not want to train panoptic stuffthingmaps? I only want to train on train and val COCO sets. If I set seg_prefix=None, it will fail as it expects a string or path. How do I disable loading these panoptic thing maps, all together?

I am not sure if it's even possible to train native COCO (not COCO-stuff) without fixing that issue.

[cailk]

Hi, sorry for the late reply.
First, our released checkpoints on COCO were all trained on only bounding-box and mask annotations. These COCO-stuff semantic segmentation annotations were not used in our models since we already disabled the semantic seg stream in model configuration "configs/base/models/htc_without_semantic_swin_fpn.py".
And if you want to fine-tune these pre-trained models on COCO on your own, you can try to delete 'gt_semantic_seg' in line 71 of "configs/BigDetection/cbnetv2/htc_cbv2_swin_base_adamw_20e_coco.py", or directly download the COCO stuffthingsmap data and move it to your dataset saving path.

[titusss]

Okay, looks like I managed to skip panoptic segmentations by following your recommendation as well as removing the semantic roi_head in the same config and setting "with_seg" to False for LoadAnnotations. All my segments are stored in mask annotations, rather than binary masks.
The "configs/base/models/htc_without_semantic_swin_fpn.py" does not seem to overrule "configs/BigDetection/cbnetv2/htc_cbv2_swin_base_adamw_20e_coco.py", so I'm not sure what you meant here. If I want to finetune BigDet CBnet on COCO, I need the "configs/BigDetection/cbnetv2/htc_cbv2_swin_base_adamw_20e_coco.py" config, correct?

Please let me know if removing that roi_head was wrong.

So for configs/BigDetection/cbnetv2/htc_cbv2_swin_base_adamw_20e_coco.py in line 8

model = dict(
    backbone=dict(
        type="CBSwinTransformer",
        embed_dim=128,
        depths=[2, 2, 18, 2],
        num_heads=[4, 8, 16, 32],
        window_size=7,
        ape=False,
        drop_path_rate=0.3,
        patch_norm=True,
        use_checkpoint=True,
    ),
    neck=dict(type="CBFPN", in_channels=[128, 256, 512, 1024]),
    test_cfg=dict(
        rcnn=dict(
            score_thr=0.001,
            nms=dict(type="soft_nms"),
        )
    ),
)

and line 58

dict(type="LoadAnnotations", with_bbox=True, with_mask=True, with_seg=False),

Also, how long does a typical training run take on 8x A100s? How many epochs did you need until the COCO model was performing well?

[titusss]

training worked well with above method! thanks

[cailk]

training worked well with above method! thanks

Good! Thank you for the feedback. As for the training time, it cost us about 12 epochs (which is 1X schedule in the Mask RCNN framework) to get a well-performing model.