Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Detectnet KITTI training doesn't seem to converge #1171

Open
rtrahms opened this issue Oct 16, 2016 · 3 comments
Open

Detectnet KITTI training doesn't seem to converge #1171

rtrahms opened this issue Oct 16, 2016 · 3 comments

Comments

@rtrahms
Copy link

rtrahms commented Oct 16, 2016

Hi all -
I have NVcaffe 0.15.13, CUDA 8.0 and CUDNN 5.0. Running Digits 4.1, and have been following the Detectnet tutorial on github. I have a GTX 1080, and the training runs, but doesn't seem to converge. mAP is always zero. Here is a screenshot of training (note this only goes through epoch 5, but I have run through epoch 30 in previous jobs - same results. The settings are identical to the tutorial as far as I can tell. What am I missing?
Thanks,
Rob

screenshot from 2016-10-16 11 10 34

@rtrahms
Copy link
Author

rtrahms commented Oct 16, 2016

I am using advice from #1108 and adjusting probabilities in the data_augmentation_layer to zero and retraining.

@rtrahms
Copy link
Author

rtrahms commented Oct 20, 2016

Setting all probabilities to zero in the data augmentation layer kicked in some learning, but mAP didn't above 20%, even after 30 epochs. Still doesn't resemble the training from the tutorial. Interestingly, testing with one of the KITTI images does pull out cars, but nothing else (no pedestrians, trucks, etc). Are there other settings I should be looking at?

@rtrahms
Copy link
Author

rtrahms commented Oct 23, 2016

I discovered in the docs that DetectNet is actually a single class object detection net:
https://github.com/NVIDIA/DIGITS/blob/digits-4.0/digits/extensions/data/objectDetection/README.md#custom-class-mappings
So I think it is working as designed, albeit not the way I was expecting. Oh well. I look forward to a multiclass version of DetectNet sometime soon!

Thanks,
Rob

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants