Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update MOT17 metrics and add MOT20 results #202

Merged
merged 4 commits into from Oct 14, 2022

Conversation

facundo-lezama
Copy link
Collaborator

@facundo-lezama facundo-lezama commented Oct 7, 2022

This PR recalculates MOT Challenge's metrics, more specifically it updates MOT17 metrics and adds MOT20 metrics. The biggest change in performance comes from using ByteTrack's YOLOX object detection model to get the detections instead of using the original detections from MOT Challenge. This is a common practice in top-ranked tracking libraries since the original detections aren't precise.

Results are reported on the train split since we don't have the ground truth for the test split. Test results must be obtained by uploading the tracking output to MOT Challenge. We are going to obtain them in a near future and update the results then.

Changes in this PR include:

  • Adding the camera motion feature to the motmetrics demo as we noticed that some sequences have camera movement.
  • Adding MOT20 challenge to motmetrics demo.
  • Using ByteTrack's YOLOX detections instead of the original ones.
  • Simplify the motmetrics test to use the new predefined distances.
  • Update README's MOT metrics.
    • MOT17 metrics show fewer rows. This is because originally we had 3 different sets of detections per video, but now that we find our own detections it doesn't make sense to process 3 times each video.

We decided not to change the motmetrics test to include MOT20 or the new detections since the overall result isn't important, we are only checking if everything remained the same. We also decided not to add camera motion to the test since this implied using the entire dataset with images and it would be too heavy to download in CI.

@facundo-lezama facundo-lezama marked this pull request as ready for review October 7, 2022 19:15
README.md Outdated Show resolved Hide resolved
@facundo-lezama facundo-lezama force-pushed the recalculate-motmetrics branch 2 times, most recently from bd7bf31 to 51a65f8 Compare October 12, 2022 17:24
Copy link
Collaborator

@javiber javiber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just had one question regarding the filter

distance_threshold=DISTANCE_THRESHOLD,
detection_threshold=DETECTION_THRESHOLD,
pointwise_hit_counter_max=POINTWISE_HIT_COUNTER_MAX,
hit_counter_max=HIT_COUNTER_MAX,
filter_factory=FilterPyKalmanFilterFactory(),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we still using filterpy? OptimizedFilter is faster and it shouldn't affect metrics too much.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried using OptimizedKalmanFilterFactory, but the metrics were a bit worse. We saw this when implementing OptimizedKalmanFilterFactory but we still haven't gotten to the bottom of it.

@facundo-lezama facundo-lezama merged commit 50f5607 into master Oct 14, 2022
@facundo-lezama facundo-lezama deleted the recalculate-motmetrics branch October 14, 2022 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants