Unstable training with keypoint loss

[johannes-tum]

First of all, thank you very much for your insightful paper! You combined multiple interesting tricks into a single powerful approach.

I tried applying the proposed keypoint loss for monocular 3D object detection on a custom dataset and on a custom detector. I have already run dozens of experiments now, but I always observe the same phenomen that training becomes unstable after a few epochs. As proposed by you, I applied:

• All keypoint losses including dimension, rotation, amodel_center, depth, all and uncertainty. I also performed experiments, where I excluded depth (also in the all loss)
• I clipped the uncertainty to have values of at least 0.01. I also tried runs without clipping and some without uncertainty.
• I applied your loss scales, but also experimented with lower and higher ones.
• I applied the chamfer loss for rotation. But I also tried applying the l1 loss on it.
• I tried runs with and runs without data augmentation.

The custom dataset is comparable to KITTI except that it requires to predict all rotation angles.
I also performed trainings on KITTI with the custom detector and your keypoint loss. There training was more stable, but there was no performance gain.
With training instability I mean that the network is learning for a few epochs. But, suddenly all losses start to increase (also 2D detection losses). Finally, the detector is unable to detect anything. For classical losses (e.g. amodel offset, l1 on rotation, etc.) training works very well. Computed keypoints are fine. We confirmed it by plotting.

Did you also observe something similar or training instabilities in general with the proposed keypoint loss? Do you have recommendations what to check or what might increase training stability?

[gkioxari]

Hi Johannes.

It's a bit hard for us to advise you when so many things are changing, namely the detector architecture and the dataset.

The fact that 2D detection and 3D detection are unstable on the training set is alarming and I would look closely at your dataset. Is the instability triggered by a learning rate diff (note we use warmup). Do you train with sufficiently large batch sizes, following our recipe? What happens when you turn off all 3D losses and only train with 2D?

Generally speaking it sounds like many things are changing in our model training and also you don't provide how you deviate from our recipe. I would suggest looking closer to your data and ablating various aspects of your model training. Unfortunately, we don't have the capacity to consult you on this one, but I hope my response was helpful.