Warning：Offset mean is XXX，larger than 100

[porcofly]

Thank you for open source your code, when train the model on Vimeo 90k，this warning occurred.
It first appeared after the first decay of the learning rate, and then the offset mean will become larger and larger. Have you ever encountered this situation? Is this normal in your training?

[Mukosame]

Hi @porcofly , basically this warning will not kill your training process, but you have to be careful of it --- since the Vimeo90k is only ~480 in size, having a 200 offset probably means your model doesn't converge.
I also encountered such situations before. In fact, such issues also happen to other works that use DCN. You can try some tricks like warm-up, or initialize with our pretrained weights to provide a safer start point for you model to prevent it from happening again.
Hope these can solve your questions!

[porcofly]

Hi @porcofly , basically this warning will not kill your training process, but you have to be careful of it --- since the Vimeo90k is only ~480 in size, having a 200 offset probably means your model doesn't converge.
I also encountered such situations before. In fact, such issues also happen to other works that use DCN. You can try some tricks like warm-up, or initialize with our pretrained weights to provide a safer start point for you model to prevent it from happening again.
Hope these can solve your questions!

Thanks for your reply, I changed the learning rate to 1 e-4 and train from scratch.Then this problem seem be solved