Car-Pedestrian multi class object detection DetectNet mAP and inference

[eweill]

I have been working on multi-class detection for cars and pedestrians using DetectNet, as mentioned in MCOD. I can successfully train a single-class DetectNet model for cars and pedestrians (separately); however I am unable to train the 2 class network correctly. Note that when I try to train the pedestrian network on KITTI, I am achieving great classification but the mAP value does not reflect that (it hovers around 10 while I can get upwards of 55 for cars).

With that said, when I try to use the 2 class detect net with a dataset created from KITTI using dontcare,car,pedestrian it only seems to detect cars no pedestrians at all (i.e. the mAP for class 1 remains 0 for the entire training process). I would like to perform inference on this, however with an mAP of 0, pedestrians aren't detected when using inference.

Can anyone point me in the right direction when training the 2-class network?

[varunvv]

@eweill

You need to change the .prototxt file for detecting multi classes. Refer the links below

https://github.com/NVIDIA/caffe/blob/caffe-0.15/examples/kitti/detectnet_network.prototxt
https://github.com/NVIDIA/caffe/blob/caffe-0.15/examples/kitti/detectnet_network-2classes.prototxt

[gheinrich]

It's notoriously difficult to train multi-class DetectNet however in case this helps, I found it easier to first train a single-class network (cars) and then fine-tune on the 2-class network (cars + pedestrians).

[eweill]

@gheinrich Since I have trained both a single class car and pedestrian network, I would like to try and fine-tune on a 2-class network. However, I am unable to use the weights directly due to dimensionality of the classifier layer.

I saved my 1-class car trained network as a "pre-trained model" in DIGITS. I then selected the 1-class model, and customized it by pasting in the 2-class prototxt. After selecting all my other desired parameters, I started the training and received the following error:

ERROR: Cannot copy param 0 weights from layer 'cvg/classifier'; shape mismatch. Source param shape is 1 1024 1 1 (1024); target param shape is 2 1024 1 1 (2048). To learn this layer's parameters from scratch rather than copying from a saved net, rename the layer.

Is there an easy way to convert either the model file or the weights file to "fine-tune" in the manner that you were suggesting or am I thinking about it entirely incorrectly? Thanks.

[gheinrich]

Why don't you rename cvg/classifier as suggested in the error message?

[eweill]

@gheinrich Thanks. I realized that as soon as I posted the Error message and got that fixed. I still am unable to train the 2-class network. I simply use a single class network that is trained on cars (with an mAP of 62, so it performs quite well) and try to "fine-tune it on pedestrians as the second class.

I replaced the custom network with a 2-class DetectNet, performed mean subtraction, and set other hyper parameters (learning rate, learning rate policy, batch size/accumulation, etc.) and no matter how I set the hyperparameters, class 2 remains at 0 mAP no matter how many iterations I choose to train the network. Am I missing an important piece to make the network train on the second class as well? Below are some of the current results I am getting.

[ShervinAr]

@eweill Hi there, did you manage to resolve the problem you had mentioned?
I am facing the same problem...

[eweill]

@ShervinAr I haven't resolved the problem yet. I have tried many different approaches and no matter how I create the LMDB dataset or train the model (different hyperparameters), I can't seem to get the second class to ever detect (mAP stays at 0 for 60 or 120 epoch).

[ShervinAr]

@eweill My guess is that the problem might have roots in :
1-different learning rates required for learning cars vs. pedestrians
2-insufficiency of training/val images for pedestrian detection: not all images in the training set include pedestrians and I am not sure how the network would react to this phenomenon...

[hadign20]

@eweill Hello. I have a similar problem and have a basic question. For multi class detection do I have to change the dataset first? I mean if the dataset label files has 10 class types (car, pedestrian, van, pickup, ...) do I have to delete the lines for van, pickup, ... or just setting the dontcare, car in the dataset creation in digits is enough?

[deagarwa]

Hello All,

I used the "detectnet_network-2classes.prototxt" file to train my network on Kitti data for cars and pedestrians.
But I am getting the following error while training:
<<
error code -11
Train net output #0: loss_bbox = 1.12773 (* 2 = 2.25547 loss)
Train net output #1: loss_coverage = 4.84529 (* 1 = 4.84529 loss)
Iteration 1264, lr = 0.0001
Snapshotting to binary proto file snapshot_iter_1276.caffemodel
Snapshotting solver state to binary proto file snapshot_iter_1276.solverstate
Iteration 1276, Testing net (#0)
Ignoring source layer train_data
Ignoring source layer train_label
Ignoring source layer train_transform

Any suggestions?

Thanks
Deepika

[lbin]

Training multi-class network based on DetectNet is not very easy, recently I got some better results. I will share some tips on how to train this type network.

[RomanSteinberg]

@lbin have you published your tips?