SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking

[johnnynunez]

Search before asking

• I have searched the YOLOv8 issues and found no similar feature requests.

Description

article: https://ai-scholar.tech/en/articles/object-tracking/smiletrack
Despite recent progress in Multiple Object Tracking (MOT), several obstacles such as occlusions, similar objects, and complex scenes remain an open challenge. Meanwhile, a systematic study of the cost-performance tradeoff for the popular tracking-by-detection paradigm is still lacking. This paper introduces SMILEtrack, an innovative object tracker that effectively addresses these challenges by integrating an efficient object detector with a Siamese network-based Similarity Learning Module (SLM). The technical contributions of SMILETrack are twofold. First, we propose an SLM that calculates the appearance similarity between two objects, overcoming the limitations of feature descriptors in Separate Detection and Embedding (SDE) models. The SLM incorporates a Patch Self-Attention (PSA) block inspired by the vision Transformer, which generates reliable features for accurate similarity matching. Second, we develop a Similarity Matching Cascade (SMC) module with a novel GATE function for robust object matching across consecutive video frames, further enhancing MOT performance. Together, these innovations help SMILETrack achieve an improved trade-off between the cost ({\em e.g.}, running speed) and performance (e.g., tracking accuracy) over several existing state-of-the-art benchmarks, including the popular BYTETrack method. SMILETrack outperforms BYTETrack by 0.4-0.8 MOTA and 2.1-2.2 HOTA points on MOT17 and MOT20 datasets. Code is available at this https URL

Use case

Faster than original Bytetrack.
ByteTrack is the SOTA tracker in MOT benchmarks with strong detector and a simple association strategy only based on motion information.
Motion information (IoU distance) is efficient and effective in short-term tracking, but can not be used for recovering targets after long-time disappear or conditions with moving camera.
So it is important to enhance ByteTrack with a ReID module for long-term tracking, improving the performance under other challenging conditions, such as moving camera.

Additional

#6485
Paper: https://arxiv.org/abs/2211.08824
Repository: https://github.com/pingyang1117/SMILEtrack_Official

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @johnnynunez, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@johnnynunez hi there! Thank you for bringing the SMILEtrack method to our attention. It sounds like a valuable contribution to the field of Multiple Object Tracking (MOT), especially with its focus on tackling the challenges of occlusions, tracking similar objects, and managing complex scenes.

While SMILEtrack is indeed an intriguing framework that offers enhanced capabilities for MOT, I'd like to note that our efforts are currently concentrated on the YOLOv8 repository and its associated features. Your approach, integrating a Similarity Learning Module with detection, complements the tracking strategies well, but integration of such a method would require careful consideration and testing to align with the existing architecture and design principles of YOLOv8.

That being said, we always welcome contributions from the community, and I appreciate your willingness to help by submitting a PR. For the best chance of your contribution being incorporated, I recommend ensuring that it fits seamlessly with the current YOLOv8 codebase, adheres to our coding standards, and is accompanied by thorough testing to prove its effectiveness.

If you have not done so already, please familiarize yourself with the specifications for contributing to our repo, which you can find in our docs. Detailed instructions will guide you on how to prepare your PR, including the necessary documentation and testing.

Keep in mind, while YOLOv8 provides a robust foundation for object detection and tracking, the implementation of additional features like SMILEtrack's Similarity Learning Module should be pursued in a manner that maintains the efficiency, simplicity, and performance that users expect from Ultralytics models.

I look forward to seeing your PR and engaging with your ideas further. Thanks again for your interest and initiative in enhancing object tracking technologies in conjunction with YOLOv8. 🚀

[johnnynunez]

Where can I upload this .pt?
https://drive.google.com/file/d/1RDuVo7jYBkyBR4ngnBaVQUtHL8nAaGaL/view?usp=share_link

[glenn-jocher]

@johnnynunez hello! It appears you have a trained model file (.pt) that you'd like to share or utilize. To clarify, if you're looking to contribute a pre-trained YOLOv8 model to the Ultralytics repository, it's best done through a pull request (PR). However, before you upload any file or submit a PR, it's essential to ensure that the model aligns with the repository's guidelines, such as relevance, performance, and licensing.

Here's what you can do:

1. Documentation: Ensure you have complete documentation for your model, including the training dataset, configuration, and any modifications made to the standard YOLOv8 architecture or training procedure.

2. Testing: Your model should be thoroughly tested to verify its performance. Include details like accuracy metrics, inference times, and compatibility with YOLOv8's existing architecture.

3. Licensing: Verify that the dataset you used for training is compatible with the AGPL-3.0 license used by the Ultralytics YOLOv8 repository.

Assuming you have followed the above steps and are ready to submit your model, you can draft a PR including all relevant data and files. However, please do not upload large files directly to GitHub. Instead, you should host the .pt file on a cloud service like Google Drive (as you have done), include a link to the file in your PR, and provide any necessary access instructions.

For more detailed submission guidelines and PR instructions, please refer to the documentation on our website. If your contribution is just for personal sharing, and not for inclusion in the Ultralytics repository, you might consider sharing the Google Drive link in forums or communities where other machine learning enthusiasts and researchers might benefit from your work.

Lastly, if your intent is different, such as needing assistance with model deployment or integration, please provide more context so we can give you the most appropriate advice. 🌟

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐