Feature: Offline Tracker (KSP Tracker) #89
Conversation
Ashp116
requested review from
soumik12345,
SkalskiP and
onuralpszr
as code owners
|
|
There was a problem hiding this comment.
Hi @Ashp116, thanks for the PR!
I have mentioned some feedback regarding the accuracy of the implementation as per the paper and also regarding how the API for using a offline tracker should be like.
| from dataclasses import dataclass | ||
| from typing import Any, Dict, List, Union | ||
|
|
||
| import networkx as nx |
There was a problem hiding this comment.
Please add networkx as a optional dependency under ksptracker. Simply run the following command and it should update pyproject.toml and uv.lock
uv add networkx --optional ksptracker| self.detection_buffer.append(detections) | ||
| return detections | ||
|
|
||
| def _calc_iou( |
There was a problem hiding this comment.
The box_iou_batch function from supervision seems like a drop-in replacement from this.
| paths.append(path[1:-1]) | ||
| return paths | ||
|
|
||
| def process_tracks(self) -> sv.Detections: |
There was a problem hiding this comment.
IMO, process_tracks should return a list[sv.Detections] where a list of tracklets are mapped to the corresponding frame of the video.
|
|
||
| # Assign track IDs | ||
| assignments = {} | ||
| for track_id, path in enumerate(paths, start=1): |
There was a problem hiding this comment.
Since this loop takes a significant amount of time to execute, please wrap the enumerate with a tqdm progress bar. For example, like the following:
from tqdm.auto import tqdm
for track_id, path in tqdm(enumerate(paths, start=1), total=len(paths), desc="Processing tracks"):| for node in path: | ||
| assignments[(node.frame_id, node.detection_id)] = track_id | ||
|
|
||
| return |
There was a problem hiding this comment.
As I mentioned in my previous comment, please make sure process_tracks returns a list[sv.Detections] where a list of tracklets are mapped to the corresponding frame of the video. I think it misses the _update_detections_with_tracks call to apply the track IDs to the detections and return the final result.
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class TrackNode: |
There was a problem hiding this comment.
It seems like this implementation uses the detections themselves as graph nodes, while the paper proposes a graph where nodes represent a grid of discrete physical locations at every point in time which allows the tracker to interpolate trajectories through areas with missing detections.
| List[List[TrackNode]]: List of paths, each path is list of TrackNodes | ||
| """ | ||
| paths: List[List[TrackNode]] = [] | ||
| for path in nx.shortest_simple_paths(graph, "source", "sink", weight="weight"): |
There was a problem hiding this comment.
nx.shortest_simple_paths generates all paths without repeated nodes between a source and target in order of increasing length, but that does not necessarily guarantee node-disjoint paths. This means that a single detection could be assigned to multiple tracks.
| assignments = {} | ||
| for track_id, path in enumerate(paths, start=1): | ||
| for node in path: | ||
| assignments[(node.frame_id, node.detection_id)] = track_id |
There was a problem hiding this comment.
This simply overwrites the track_id, meaning only the last path found containing that detection gets the assignment. This doesn't seem like an accurate representation of the disjoint-path formulation described in the paper.
Description
This PR introduces a new offline Multiple Object Tracker (MOT) based on the K-Shortest Paths (KSP) optimization algorithm. This tracker is based on the method proposed in the paper: "Multiple Object Tracking using K-Shortest Paths Optimization" by Jérôme Berclaz et al. The tracker models detections across frames as a directed acyclic graph and solves for globally optimal object trajectories by computing disjoint shortest paths.
The inspiration for this PR: #59
List any dependencies that are required for this change.
Yes, this PR needs
networkxto build a directed acyclic graph.Type of change
How has this change been tested? Please provide a testcase or example of how you tested the change?
This PR can be tested using the Colab notebook provided here. The notebook can be used to test KSP Tracker. Currently, the notebook is using one of the MOT17 Challenge Datasets, specifically MOT17-02-FRCNN.
Any specific deployment considerations
N/A
Docs