Tracker

[dbroboflow]

Description

This commit includes updates to the repository's main README.md file, providing usage details, and explanations for the example code located in the "examples" folder. The updates aim to improve the documentation and guide users on how to utilize the Roboflow Computer Vision Library for object detection and tracking.

Dependencies required for this change:
Dependencies specified in setup.py

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

The code for the example "Object Tracking with Byte Track" was tested on a sample video and successfully performed object detection and tracking. The test case includes verifying the correct detection of objects, assigning unique tracker IDs to each object, and generating the output video with annotated bounding boxes.

Docs

• Docs updated? What were the changes:

Bytetrack added under the supervision/tracker folder.

Updated the "Examples" section to include instructions for running the "Object Tracking with Byte Track" example code under the "examples" folder.

[github-actions]

Hello there, thank you for opening an PR ! 🙏🏻 The team was notified and they will get back to you asap.

[SkalskiP]

Hi @dbroboflow, I'm currently reviewing other PRs. There is a lot of work happening in parallel. I will do my best to check it today.

[SkalskiP]

Hi @dbroboflow 👋🏻

• Could you run the commands below before you submit the PR?

  isort --profile black supervision/
  black supervision
  

• Our goal is to improve on the original implementation. Make it easier to use and cleaner.

• Take a look at changes dine by YOLOv8 team here. That is good starting point.

• We use PEP8 as the naming convention standard. Please keep the names of the packages in the snake case. ByteTracker -> byte_tracker.

• Please make sure to follow the API defined in Tracking support #195

• Please ensure that every function and method use type hints for input and output.

[hardikdava]

Hello @SkalskiP and @dbroboflow, I saw the initial implementation. My thoughts after reviewing the implementation are as following.

It would be easy to use something like

tracker = sv.ByteTracker()

with VideoSink(TARGET_VIDEO_PATH, video_info) as sink:
    # loop over video frames
    for frame in tqdm(generator, total=video_info.total_frames):
        dets = sv.Detections(...)
        tracked_dets = tracker.update(dets)
       # Annotate tracked detections....

curent implementation has too much overhead for example, detections2boxes() and again matching detections to get tracking id will make tracker slower. The tracker part should handle class id and tracker id, as a user I should not be worried about intermediate functions.

byte_track.py

class ByteTrack:
    ...
    def update(detections: sv.Detections) -> sv.Detection:
          ....
          return tracked_detections

If we do this then when there is a new tracker, it can also be implemented easily with same structure. Tracker API usage will be same in the future also.

[SkalskiP]

@hardikdava 100 agree. Take a look at #195, The design I proposed is matching how you see stuff as well.

[dbroboflow]

@hardikdava 100 agree. Take a look at #195, The design I proposed is matching how you see stuff as well.

We are trying to pass arrays on the inference server. We have some bottlenecks. Also we want to remove object conversion, and pass to np array directly.

Current version:

Decoding --> Create Infer Object --> Inference --> Response Object

Plan:

Decoding --> Queue(np array) --> Preprocessing --> Queue(np array) --> Inference(np array) --> Result 
                                                                                                                                                                 --> Tracker 

I can add

def update(detections: sv.Detections) -> sv.Detection:

But also we want to keep original bytetrack input. If it works, I can do it as optional.

Thank you,

[artyaltanzaya]

Suggestion: The class variables track_id, is_activated, state, history, etc., will be shared across instances of BaseTrack. If they are supposed to be different for each instance, they should be instance variables (i.e initialized in a init method).

 def __init__(self):
        self.track_id = 0
        self.is_activated = False
        self.state = TrackState.New
        self.history = OrderedDict()  
        self.features = []
        self.curr_feature = None
        self.score = 0
        self.start_frame = 0
        self.frame_id = 0
        self.time_since_update = 0
        self.location = (np.inf, np.inf)  # Coordinates in a multi-camera setup

[SkalskiP]

Great point @artyaltanzaya! We absolutely want to make those tied to instance not a class.

[dbroboflow]

I updated as in the comment. But rn we are creating new object for each camera. Do you need some features for tracker like add camera(camera_link), remove camera(camera_id), update(self, output_results, img_info, img_size, camera_id) ?

[SkalskiP]

Nope. The whole camera abstraction should be on the inference server side.

[artyaltanzaya]

The file yolov5s.pt is directly used in the script. We might want to check if this file exists before running.

[SkalskiP]

I don't want us to have that script inside the project. We should aim for a similar format of documentation as with all other supervision features. Take a look here: https://roboflow.github.io/supervision/dataset/core/

[dbroboflow]

It is ultralytics repo and it downloads automatically the pre-trained model. I added that one as a sample. We can remove the test files and add to another doc if you want.

[SkalskiP]

Hi @dbroboflow 👋🏻! We don't store example scripts in supervision. But we document API, just like in the link I provided. If you look there, we also have examples in the docs. We just format those in a different way.

[dbroboflow]

Hi @SkalskiP! I updated the tracker as update_from_numpy and update_from_detections. When we are merging it we can delete the samples. I don't delete these files, when you tested it again. I will remove it.

[hardikdava]

@SkalskiP let me know once it ready to test. I can do that and also review it together.

[dbroboflow]

@SkalskiP let me know once it ready to test. I can do that and also review it together.

@hardikdava I updated the repo. Also I tested it and checked the outputs. I think we can do review. Also, if you will set a meeting, I would like to join when you are reviewing it.

[dbroboflow]

@dbroboflow, let's do it! I'm waiting for your API updates. @hardikdava, thanks a lot for coming up with ideas! 🔥

Notice that you currently have conflicts with main. Please resolve it if you have an opportunity as well.

And equally important, ensure that your docstring is formatted consistently with te rest of the codebase. It will allow us to automatically generate documentation.

Hmm I just touch the setup.py file which you will delete in the future. I will check the conflicts. Also I added the similar docs comment for docs generation. Which tool are you using like Sphinx?

[SkalskiP]

Hmm I just touch the setup.py file which you will delete in the future.

We won't be able to merge if you have conflicts. So please resolve them before we start the review.

Also I added the similar docs comment for docs generation. Which tool are you using like Sphinx?

We use MKDocs. You can check if everything works. Just run mkdocs serve in the terminal.

[SkalskiP]

Hi, @dbroboflow 👋🏻!

• The fact that you updated 20 files makes it super hard to do the review. At this point, is there any way we could clean it up? Especially since I don't want to make many of those formatting changes.

• I see you managed to plug bytetrack.md into docs. It is awesome that you added some information about tracker usage, but we lack API reference information:

  • As I mentioned. Your docstrings are in the incorrect format and won't be parsed by our documentation framework. Please take a look at the rest of the code, and use it as a reference.
  • When you will change the format, you'll be able to use include API reference like this: :::supervision.tracker.byte_track.core

• Please resolve merge conflicts

• Add ByteTrack import to supervision.__init__.py. So supervision could use ByteTrack like this:

  import supervision as sv
  
  tracker = sv.ByteTrack(
        track_thresh=0.25, 
        track_buffer=30, 
        match_thresh=0.8, 
        aspect_ratio_thresh=3.0, 
        min_box_area=1.0, 
        mot20=False, 
        frame_rate=30
     )

[SkalskiP]

Reformat docstrings to format used in other parts of the codebase.

[SkalskiP]

Hi, @artyaltanzaya 👋🏻! Could you take a look at the PR and confirm it builds, installs, and work as expected in the issue: #195?

[artyaltanzaya]

@SkalskiP it looks like you left comments recently, were those incorporated? should I wait until the changes are made to run tests?

[hardikdava]

@dbroboflow We can utilize sv.box_iou_batch to find intersection over union between (N, 4) and (M,4) boxes. This method is much faster and implemented in vectorized. You implementations will be slower by utilizing for loops. Please check it out.

[hardikdava]

@SkalskiP @dbroboflow import ultralytics package might create an issue for license. Please verify it once.

[dbroboflow]

These are just for test scripts. When we are merging it, we will delete it. Also I added it to the docs. We don't need to install ultralytics package.

[hardikdava]

@dbroboflow @SkalskiP Please add lap or scikit based on usage.

[hardikdava]

@SkalskiP @onuralpszr Can we make installation like pip install supervision[tracker] and it will install dependency for tracker otherwise use normal installation. Installation of scikit-learn might create an issue for arm devices. What do you think?

[onuralpszr]

I can see "arm64" support for scikit-learn

https://pypi.org/project/scikit-learn/1.3.0/#files

"arm" (32-bit) already consider old. And If you buy a new arm based board, it will be high likely arm64 devices so it is fine on that as well any other package worry about ?

[hardikdava]

@dbroboflow Why do we have to match detections and tracklets again?

[hardikdava]

@dbroboflow Just try to convert tracks to sv.Detections format and you should be fine.

[dbroboflow]

Hi Hardik,

We will use it in the inference server. Because of that, there are separated methods.

[hardikdava]

Okay. Understood. But still in the method of update_from_detection method you are matching detection and tracks again which is unnecessary part. I meant you can just convert result of update_from_numpy to sv.Detection format.

[SkalskiP]

Closing this PR. @dbroboflow opened a new clean one here: #256