The Training Loss Doesnt Decrease at All

[christofel04]

Hello Wang and friends,

great work on 3D occupancy prediction using 4D radar.

Actually I try to train 3D occupancy prediction using 4D radar dataset but my training loss doesnt decrease at all as below:

I am using K-Radar Sparse Tensor using 8 Dopplers generated with code "generated_4d_polar_doppler".

But I got warning error that my input feature doesnt have gradient None as below.

Is this warning error make training 3D occupancy prediction using 4D radar cannot be trained?

[Ericwen2001]

Hi christofel04, I believe this behavior is due to the loss normalization applied during training.

As seen in the code

RadarOcc/projects/occ_plugin/occupancy/detectors/radarocc_self_small.py

Line 506 in 1bc1c3f

losses[loss_key] = losses[loss_key] / (losses[loss_key].detach() + 1e-9)

here, loss normalization is commonly used in the community to balance multiple loss functions.
The decrease in grad_norm suggests that training is progressing as expected. Regarding the warning, it’s hard to determine the cause just from this screenshot—it might be safe to ignore, but additional context would help confirm.

[christofel04]

Hello Eric and teams,

thank you for prompt answer.

It make sense because actually I have problem when using Lovasz loss hence I dont use Lovasz loss as below:

Then I use normal segmentation loss as below in Lovasz function.

Do you know how to solve problem with Lovasz loss? Because I dont have any problem when installing torch and cuda toolkits.

[YueSun0609]

I met the same problem