Training is stuck for pretraining

[mu-cai]

I found an error around line 200 in transforms.py. For some samples, it never goes out of the loop, which makes the training not work, meaning that len_indexes= sum_indexes=0

[CSautier]

Hi, thank you for the notice.
Did you experiment the bug in practice on a nuScenes training? Because while it's true that the code might not work on some specific conditions, the resizecrop has a crop ratio reaching 1, and allowing for the original image ratio, thus the resizecrop has a non-zero probability of outputting the unmodified input image, which should be non-blocking, unless there is some problem that I do not see. In all my pre-training experiments on nuScenes, I never had an issue of this transform being stuck indefinitely.

[mu-cai]

Thank you for your Response!
However, at a certain sample, specifically, idx = 17903, camera_id =3, the mask could be all zero:

[CSautier]

I just did a re-train on a few epochs, with no issue of workers stucking there. Have you modified anything in the config? Are you running with slidr_minkunet.yaml?
I've looked specifically at the dataset index 17903 and I don't see that there would be a fully empty mask. mask.sum() gives me 1115 with camera_id=3 before any augmentation. Then there is the DropCuboid which cannot reduce that lower than 1024 (line 252) meaning that on line 150 where I assume you put your breakpoint, there is no reason for the mask to be empty, unless it was empty from the start (which isn't the case for me), then this might either be corrupted data on your dataset, or a modification you did to dataloader_nuscenes.py.

[mu-cai]

Thank you! I just found that the images are corrupted! Thank you for the notice! Really helpful!
Besides, I wonder for the downstream task of object detection, do you just transfer the pretrained weights in your repo to OpenPCdet? If so, how do you conduct the experiments on "Results on the validation set using Minkowski SR-Unet with a fraction of the training labels"?
Thank you!

[CSautier]

Hi, I'm sorry for the late reply.
Glad I could be of help; as for the Minkowski SR-Unet, we modified PointRCNN to use as backbone our Minkowski-UNet, where we have loaded the weights. We also modified the kitti dataloader to only keep 1 in n training data to use a fraction of the training data. This can be done by subsampling self.kitti_infos in kitti_dataset.py of OpenPCDet.
We did not provide the code as it did not belong to us, and that would have added an entire fork just for this one experiment, but I can send it to you by email if you need it.

[mu-cai]

Thanks! It will be great if you can send me the code via email!

…

Sent from my iPhone

On Jul 24, 2022, at 2:39 AM, CSautier ***@***.***> wrote:  Hi, I'm sorry for the late reply. Glad I could be of help; as for the Minkowski SR-Unet, we modified PointRCNN to use as backbone our Minkowski-UNet, where we have loaded the weights. We also modified the kitti dataloader to only keep 1 in n training data to use a fraction of the training data. This can be done by subsampling self.kitti_infos in kitti_dataset.py of OpenPCDet. We did not provide the code as it did not belong to us, and that would have added an entire fork just for this one experiment, but I can send it to you by email if you need it. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[CSautier]

I sent you the files by email and am closing this issue. Let me know by mail if you need help to use it.

[mu-cai]

I wonder whether there is a bug around line https://github.com/valeoai/SLidR/blob/main/pretrain/dataloader_nuscenes_spconv.py#L321? (You should not comment this line) Because in your object detection code base, you use 4 features in object detection.