Code clarification

[johnwlambert]

Hi @xinshuoweng , thanks for open-sourcing your 3d tracking code. Very nice to find this. I have a few questions for you about the coordinate system and dimension ordering:

1. The docstring here doesn't look correct -- mentions image projection.
   https://github.com/xinshuoweng/AB3DMOT/blob/master/main.py#L115

2. It looks like the tracking reference frame is the camera coordinate frame, rather than an egovehicle, LiDAR, or world frame. Might be helpful to clarify this in the docstrings. Unlike KITTI, most new datasets now annotate in an egovehicle, LiDAR, or world frame, instead of the camera frame, meaning x is now associated with length, y with width, and z with height, e.g. Argoverse or Nuscenes
   https://github.com/xinshuoweng/AB3DMOT/blob/master/main.py#L134

3. What is the reason for the reordering back and forth of the bounding box information? Might be helpful to indicate the order expected (since this threw me off initially)
   https://github.com/xinshuoweng/AB3DMOT/blob/master/main.py#L348

4. Any particular reason to not use Shapely.geometry for polygon intersection? I see that you've provided your own function here:
   https://github.com/xinshuoweng/AB3DMOT/blob/master/main.py#L48

[johnwlambert]

Asking about the polygon intersection because seeing:

AB3DMOT/main.py:56: RuntimeWarning: divide by zero encountered in double_scalars
  n3 = 1.0 / (dc[0] * dp[1] - dc[1] * dp[0])
AB3DMOT/main.py:57: RuntimeWarning: invalid value encountered in double_scalars
  return [(n1*dp[0] - n2*dc[0]) * n3, (n1*dp[1] - n2*dc[1]) * n3]
AB3DMOT/main.py", line 397, in update
    matched, unmatched_dets, unmatched_trks = associate_detections_to_trackers(dets_8corner, trks_8corner)
AB3DMOT/main.py", line 321, in associate_detections_to_trackers
    iou_matrix[d,t] = iou3d(det,trk)[0]             # det: 8 x 3, trk: 8 x 3
AB3DMOT/main.py", line 98, in iou3d
    inter, inter_area = convex_hull_intersection(rect1, rect2)
  File "qhull.pyx", line 2359, in scipy.spatial.qhull.ConvexHull.__init__
  File "qhull.pyx", line 279, in scipy.spatial.qhull._Qhull.__init__
ValueError: Points cannot contain NaN

[johnwlambert]

You could instead use:

from shapely.geometry import Polygon

def shapely_polygon_intersection(poly1, poly2):
  """
  """
  poly1 = Polygon(poly1)
  poly2 = Polygon(poly2)
  return poly1.intersection(poly2).area


def test_shapely_polygon_intersection1():
  """
  """
  poly1 = np.array(
    [
      [0,0],
      [3,0],
      [3,3],
      [0,3]
    ])
  poly2 = np.array(
    [
      [2,1],
      [5,1],
      [5,4],
      [2,4]
    ])
  inter_area = shapely_polygon_intersection(poly1, poly2)
  assert inter_area == 2


def test_shapely_polygon_intersection2():
  """
  """
  poly1 = np.array(
    [
      [0,0],
      [4,0],
      [4,4],
      [0,4]
    ])
  poly2 = np.array(
    [
      [0,0],
      [4,0],
      [4,4],
    ])
  inter_area = shapely_polygon_intersection(poly1, poly2)
  assert inter_area == 8
  



def iou3d(corners1, corners2):
    ''' Compute 3D bounding box IoU.

    Input:
        corners1: numpy array (8,3), assume up direction is negative Y
        corners2: numpy array (8,3), assume up direction is negative Y
    Output:
        iou: 3D bounding box IoU
        iou_2d: bird's eye view 2D bounding box IoU

    '''
    # corner points are in counter clockwise order
    rect1 = [(corners1[i,0], corners1[i,2]) for i in range(3,-1,-1)]
    rect2 = [(corners2[i,0], corners2[i,2]) for i in range(3,-1,-1)] 
    area1 = poly_area(np.array(rect1)[:,0], np.array(rect1)[:,1])
    area2 = poly_area(np.array(rect2)[:,0], np.array(rect2)[:,1])

    inter_area = shapely_polygon_intersection(rect1, rect2)

    #inter, inter_area = convex_hull_intersection(rect1, rect2)
    iou_2d = inter_area/(area1+area2-inter_area)
    ymax = min(corners1[0,1], corners2[0,1])
    ymin = max(corners1[4,1], corners2[4,1])
    inter_vol = inter_area * max(0.0, ymax-ymin)
    vol1 = box3d_vol(corners1)
    vol2 = box3d_vol(corners2)
    iou = inter_vol / (vol1 + vol2 - inter_vol)
    return iou, iou_2d

[xinshuoweng]

HI @johnwlambert, I have updated some comments in the code that you mentioned for further clarification. Thank you for bringing those points.

Also, I have checked your code, which looks great. The only downside I found now is that the function of "shapely_polygon_intersection" takes a longer time than "convex_hull_intersection". As speed is crucial to multi-object tracking. I am not sure if it is good to use the new function.

For the error brought by the convex_hull_intersection, it can be simply resolved by adding "try exception" in the code.