Detecting multiple classes

[KevinTang29]

Hello @happinesslz , thanks for the great code. I would like to detect multiple classes using EPNet. I modified your code based on the modifications in #3 except the last one. For the last one, I changed

EPNet/lib/net/train_functions.py

Line 236 in 0123c34

loss_utils.get_reg_loss(torch.sigmoid(rcnn_cls_flat)[fg_mask], mask_score[fg_mask],

to loss_utils.get_reg_loss(torch.max(F.softmax(rcnn_cls_reshape, 1), 1)[0][fg_mask], mask_score[fg_mask],
In addition, after this line

EPNet/lib/rpn/proposal_target_layer.py

Line 69 in 0123c34

batch_cls_label = (batch_roi_iou > cfg.RCNN.CLS_FG_THRESH).long()

I added
batch_cls_label[batch_cls_label == 1] = batch_gt_label_of_rois[batch_cls_label == 1]
to make the label of proposals to be the class label of their corresponding ground truth boxes(1~n) or background(0) instead of 0/1 (and also several other changes to read the labels of gt boxes when reading data and store them in batch_gt_label_of_rois, and several changes in the postprocessing after model inference)
After these changes the training process seems to work. However, when I tested the trained model, I found that the model tends to predict every proposal to be the background class. I changed the predicted class to be the class with the max softmax value except the background class and greater than a certain threshold (say 0.05), and the result seems ok. So I guess it's caused by the imbalance between the background class and other classes in training. I tried to increase the number

EPNet/tools/cfgs/LI_Fusion_with_attention_use_ce_loss.yaml

Line 135 in 0123c34

FG_RATIO: 0.5

to 0.75 and train again but seems no use. Could you please give me some advice on this? Thank you!

[happinesslz]

@KevinTang29 Maybe Focal loss is an available method to solve the imbalance between the background class and other classes in training. Please have a try, and use "SigmoidFocalLoss" instead of "CrossEntropy" for multi-classes in the LI_Fusion_with_attention_use_ce_loss.yaml. Good luck to you!

[KevinTang29]

@happinesslz Thank you very much for your advice. It seems that the original "SigmoidFocalLoss" can only be applied in binary classification, and I tried to change it to focal loss under multiple classification and trained with the focal loss. However, the problem still exists. Maybe the parameter alpha and gamma need to be tuned under multiple classification? In addition, I tried to increase the cross entropy weights of the object classes

EPNet/tools/cfgs/LI_Fusion_with_attention_use_ce_loss.yaml

Line 130 in 0123c34

CLS_WEIGHT: [1.0, 1.0, 1.0]

to [1.0, 2.0, 2.0, 2.0], and also decrease the roi number in training

EPNet/tools/cfgs/LI_Fusion_with_attention_use_ce_loss.yaml

Line 136 in 0123c34

ROI_PER_IMAGE: 64

to 32 and trained with CE loss again, and the result is better. (though still not right, something like the classfication score changed from [0.9, 0.08, 0.01, 0.01] to [0.7, 0.25, 0.03, 0.02] if the roi is actually of class 1). Is this another possible way to solve the problem?

[Benedict0819]

Did you make it work for multiclass-Epnet?

[KevinTang29]

Did you make it work for multiclass-Epnet?

I just trained the model with CE loss with class weights. I set the model to detect four classes ['Car', 'Pedestrian', 'Cyclist', 'Truck'] and set the corresponding weights to be [1.0, 3.0, 6.0, 6.0, 12.0] (the first weight is of background class). The evaluation results seemed acceptable, though not good enough. Maybe the weights and other parameters need to be tuned further.

[Benedict0819]

I have finished , can you show your mail to me ?and we can talk deeper ------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "happinesslz/EPNet" <notifications@github.com&gt;; 发送时间:&nbsp;2021年1月28日(星期四) 下午4:03 收件人:&nbsp;"happinesslz/EPNet"<EPNet@noreply.github.com&gt;; 抄送:&nbsp;"何必歇斯底里"<1481706330@qq.com&gt;;"Comment"<comment@noreply.github.com&gt;; 主题:&nbsp;Re: [happinesslz/EPNet] Detecting multiple classes (#8) Did you make it work for multiclass-Epnet? I just trained the model with CE loss with class weights. I set the model to detect four classes ['Car', 'Pedestrian', 'Cyclist', 'Truck'] and set the corresponding weights to be [1.0, 3.0, 6.0, 6.0, 12.0] (the first weight is of background class). The evaluation results seemed acceptable, though not good enough. Maybe the weights and other parameters need to be tuned further. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[KevinTang29]

I have finished , can you show your mail to me ?and we can talk deeper ------------------ 原始邮件 ------------------ 发件人: "happinesslz/EPNet" <notifications@github.com>; 发送时间: 2021年1月28日(星期四) 下午4:03 收件人: "happinesslz/EPNet"<EPNet@noreply.github.com>; 抄送: "何必歇斯底里"<1481706330@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [happinesslz/EPNet] Detecting multiple classes (#8) Did you make it work for multiclass-Epnet? I just trained the model with CE loss with class weights. I set the model to detect four classes ['Car', 'Pedestrian', 'Cyclist', 'Truck'] and set the corresponding weights to be [1.0, 3.0, 6.0, 6.0, 12.0] (the first weight is of background class). The evaluation results seemed acceptable, though not good enough. Maybe the weights and other parameters need to be tuned further. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

I would like to know how you did. My Email is kevintang29@163.com

[TimGor1997]

I have finished , can you show your mail to me ?and we can talk deeper ------------------ 原始邮件 ------------------ 发件人: "happinesslz/EPNet" <notifications@github.com>; 发送时间: 2021年1月28日(星期四) 下午4:03 收件人: "happinesslz/EPNet"<EPNet@noreply.github.com>; 抄送: "何必歇斯底里"<1481706330@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [happinesslz/EPNet] Detecting multiple classes (#8) Did you make it work for multiclass-Epnet? I just trained the model with CE loss with class weights. I set the model to detect four classes ['Car', 'Pedestrian', 'Cyclist', 'Truck'] and set the corresponding weights to be [1.0, 3.0, 6.0, 6.0, 12.0] (the first weight is of background class). The evaluation results seemed acceptable, though not good enough. Maybe the weights and other parameters need to be tuned further. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

Hello @Benedict0819 , I also want to achevice the multi-clas on EPNet, could you please share your code about multi-class to me?
My email address is 595603009@qq.com
Thank you very much!