Fix for non-distributed training

[GerardMaggiolino]

I thought I'd document a fix for non-distributed training in case other people are trying to use this repo as well.

Even if distributed training isn't selected, you'll get a:

RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

during the forward call.

This happens in the decoder head due to the SyncBN layer. Find the config file under base corresponding to your model, and change the norm_cfg to use BN instead of SyncBN. For example, I'm using SeMask-FPN, so under configs/base/models/semfpn_semask_swin.py, I change:

norm_cfg = dict(type='SyncBN', requires_grad=True)

to

norm_cfg = dict(type='BN', requires_grad=True)

The layers seem to be totally compatible with one another, so you can load weights from models using SyncBN fine. Just a warning that you'll need to hardcode this if changing between regular and distributed training.

[praeclarumjj3]

Hi @GerardMaggiolino, thanks for your interest.

You can quickly run a process on a single GPU with the tools/dist_train.sh script by setting the GPUS=1. You won't have to hardcode anything in that case. Is there some disadvantage to doing that and running the regular training instead?

[GerardMaggiolino]

This is a great point, and I don't think there is any issue with just using the dist_train.sh script instead of running train.py directly. Thanks for the great work btw, I've used SeMask for a few projects and the pre-trained weights are really helpful!

[praeclarumjj3]

Thanks for the praise. Glad our work could assist you! 😄