Does this require synchronous sensor inputs if used in real time?

[sisaha9]

I want to try using this repo but my lidar and cameras do not publish data at the same frequency. Is this capable of running in only Lidar or only camera data to update the current latest estimates?

[aleksandrkim61]

• The framework relies on the fact that 3D and 2D inputs describe exactly the same scene, so I expect mismatched data to produce less precise tracking results. However, if the time difference is small, it should still perform very well. The problem with different sensor frequency is misalignment of 2D and 3D detection boxes during fusion and 2nd stage association - nothing else relies on sensors being synchronised. One way to accommodate this could be to expand 2D detection boxes - make them bigger in all directions, so they can be matched to 3D boxes that were captured earlier/later. Or change the IoU thresholds for fusion and 2nd association - parameters fusion_iou_threshold and leftover_matching_thres - see configs/params.py.

• If you are talking about not having both 3D and 2D for each frame, then it is possible. 3D-only is enough to update the position of a track, 2D-only is enough to keep the track alive and rely on Kalman Filter predictions. That's one of the major points of this work - expect two sources of detections to be able to update tracks with at least one of them. More details are in the paper section III-B Matching.
  Just remember that detections are consumed as a dictionary for each frame, so make sure you return empty lists for frames where only one of the sensors is used. For example, MOTSequence.load_detections_3d could return a defaultdict(list)

[aleksandrkim61]

No reply to the answer, so I assume there is no follow-up.