Question about the threshold of score filter

[zengwz]

Hi, I find the threshold of score filter is very slow, about 0.16. And there will be many FP trajectories generated as valid trajectories. But the final evaluation indicators are still good. Why do you choose such a low threshold and why the final evaluation metric was still excellent?

[lixiaoyu2000]

Hi, what you are talking about is a very interesting question. However, I think 0.16 is a high rather than a low threshold. The reasons for choosing 0.16 are as following:

• Detector. The thresholds of SF are detector-specific. In the val set, we integrate CenterPoint, a powerful 3D detector. We believe that the confidence score of 0.16 can almost determine that the object is TP.
• Pre-processing. We also introduce NMS as an addition but critical pre-processing technique, which can quickly eliminate FP caused by redundancy and ensure the high-quality of detections. (Please see the ablation experiment in the article)
• Experiment. Typically, we will perform a linear search on the validation set to obtain the most appropriate threshold.
• Others. In experiments, one of our important findings is that placing SF in the preprocessing module leads to better results than placing SF in data association module (like simpletrack, Bytetrack, etc.). One potential reason may be that artificially using threshold to limit multi-stage correlation will narrow the matching domain and lead to unreliable matching results.

If you have any further questions, please feel free to chat with me. 😊

[zengwz]

In the first picture, 82 bboxes still remains after SF and NMS in the first frame. 82 bboxes in the first frame here should directly generate new trajectories in your tracking process.

In the second picture, I visualize the gt bboxes and the number of gt bboxes is 34 in this frame. I think the key point is that the bboxes left after SF and NMS stills contain many FP bboxes(82 vs 34), which seems very confusing.

[lixiaoyu2000]

• First of all, I think that a comparison of one frame is not enough to prove that a hyperparameter suitable for all scenarios is bad.
• Secondly, AMOTA is an indicator that comprehensively considers FP, FN and IDS. According to your considerations, although raising the threshold will suppress FP, it will also bring FN. The current threshold is a trade-off between the two. You can try fine-tuning the threshold to see the results under different hyperparameters. Besides, AMOTA is an indicator that emphasizes recall rate, see here. In other words, FN will affect the final tracking results more than FP.
• Finally, in the Poly-MOT, the FP trajectory will be severely penalized, which makes FP usually only appear under a lower threshold. In other words, even though the tracking performance (MOTA) may be weak at a high recall rate, I believe that the tracking performance will be outstanding at other recall rates, which allows the average MOTA (AMOTA) to still be at a very high standard.

By the way, your visualization looks pretty. Is it possible to share this visual script? 😃

[zengwz]

Oh, I get it. thank you for your reply. And the visual script comes from https://blog.csdn.net/qq_16137569/article/details/121066977.

[lixiaoyu2000]

Nice! Any further questions, feel free to contact me. This issue will be closed.