Not able to reproduce the results listed in the paper with my trained model

[LigZhong]

I met a problem of mode collapse when step number is larger than 300K, and with the final model I got, I am not able to reproduce the result shown int the paper. Can you give your loss curve? @Paper99

[Paper99]

Due to the offset overflow of the deformable convolution, the training process may be collapsed.
Such issue also occurs in other works that use DCN.
Please refer to this issue for more details.

However, we never met this problem when we trained the final model (which takes 500K iters).
The collapsed process may lead to an undesired model, whose result is far from that of our released model.
I suggest that you could retrain it or select an early safer checkpoint to resume training.

For convenience, we provide our loss curves as follows:

I hope these could help you.

[sydney0zq]

Due to the offset overflow of the deformable convolution, the training process may be collapsed. Such issue also occurs in other works that use DCN. Please refer to this issue for more details.

However, we never met this problem when we trained the final model (which takes 500K iters). The collapsed process may lead to an undesired model, whose result is far from that of our released model. I suggest that you could retrain it or select an early safer checkpoint to resume training.

For convenience, we provide our loss curves as follows:

I hope these could help you.

@Paper99 I also encounter this problem on my machine, which GPU card did you use for training? I use V100 32G, and it will collapse at about 300k iter.

[Paper99]

Hi, we use 8 V100 (16G) GPUs or 8 1080ti GPUs to train our model.

[sydney0zq]

@Paper99 How do you suggest to solve the collapse problem? If we do clipping on the DCN module's weight, I cannot confirm the range ... Do we have another replaceable module to avoid the issue?

[sydney0zq]

@Paper99 Hello, how do you select the final release checkpoint, The 50w iter checkpoint or select the best among several final instances?

[Paper99]

Just choose the best.

[MasterHow]

same question.

[jiahui1688]

@Paper99 How do you suggest to solve the collapse problem? If we do clipping on the DCN module's weight, I cannot confirm the range ... Do we have another replaceable module to avoid the issue?

Hi, Is the problem solved? I have the same problem. Thank you.