Coordinate system of 3D bounding box annotation

[eunyoung000]

Hi there,

We are using Unity for 3D bounding box annotation but not sure about which coordinate system the generated boxes are based on.
Are they (i.e. translation & rotation of the 3d box) based on the camera coordinate system?
Thanks

[eunyoung000]

Also, below shows the camera intrinsic values we are getting and they don't look like valid values/

"camera_intrinsic": [
[
0.974278569,
0.0,
0.0
],
[
0.0,
1.73205078,
0.0
],
[
0.0,
0.0,
-1.0006001
]
],
Can you also guide us to get camera intrinsic values correctly?
Thanks

[csun]

Hey there,

For the 3D bounding box, the translation and rotation are with respect to the world coordinate frame, not the camera. (edit: this is wrong, it's in the sensor coordinate frame. See later comment)

For the camera intrinsics, they're stored as a matrix in the format described here.

Hope that helps! Please let me know if you have other questions.

[eunyoung000]

Thanks for the quick reply, but I am not sure if the bounding box is actually w.r.t. the world coordinate system.

Below show the saved output for the camera and one of the 3D box.

"sensor": {
"sensor_id": "camera",
"translation": [
-34.152092,
3.3366034,
-58.94595
],
}

"annotations": [
{
"id": "bounding box 3D",
"annotation_definition": "bounding box 3D",
"values": [
{
"label_id": 1,
"label_name": "Celebration",
"instance_id": 1,
"translation": {
"x": -0.354047179,
"y": -0.363202155,
"z": 2.176908
},

The camera position is w.r.t. the world coordinate system but the 3D box is not in the world coordinate system: it looks like it is w.r.t the camera position or something else (If it's w.r.t. world coordinate system, the translation value should be similar to the camera position, because the box was around the camera. )

Feel free to correct me if my understanding is wrong. Thanks

[eunyoung000]

Also, why do we have -1 at (3, 3) in the intrinsic matrix? The page says it is set to 1.
And, our intrinsic matrix doesn't include cx, cy.

[Robotatron]

Hey there,

For the 3D bounding box, the translation and rotation are with respect to the world coordinate frame, not the camera.

The position of the 3D bbox is labeled relative to the camera. In the example below, the camera is at (0, 0, -5) and the cube is at origin (0,0,0)

ps. why am I getting these warnings?

[csun]

Ah, that's my bad. You're right - it's in the camera coordinate frame. I was reading the docstring for this class, though it appears that this is transformed into camera space here before being output. I'll edit my original comment.

For the camera intrinsics, to be honest, I don't know. I was just going off of this documentation (which ironically contains the correct answer to your bounding box question). I can ask someone else on the team to chime in here.

[StevenBorkman]

Hey all, the camera intrinsic matrix captured in the perception camera is just a 3x3 version of the Unity Camera projectionMatrix. The 3x3 upper left corner encodes the rotation and scale of the camera. A lot of good information about projection matrices can be found here.

The main thing that we use the intrinsic matrix for is to convert coordinates from camera space to screen space. Although we are not using the camera matrix directly in our BoundingBox3DLabeler::OnVisualize() method, because we have access to the camera's transform directly, it illustrates how to convert coordinates from camera to screen space.

Finally, if you are comfortable with python code, we have datasetinsights, a python project which creates notebooks, dispalys stats, and visualizations of perception data. Here I have linked code which draws the 3D bounding box on an image. This code illustrates how to use the intrinsic matrix to convert from camera to screen space.

I hope this helps out and let me know if I or @csun can be of any more assistance.

[eunyoung000]

Cool thanks for the detailed explanation!

[clydebailey]

@StevenBorkman
Hey I am trying to convert the camera coorindate system to the pixelwise coordinate system using your above mentioned transformations. I am trying to detect cricket bats in the image.

img_path = "/data/solo/sequence.1/step0.camera.png"
img = cv2.imread(img_path)
ann_path = img_path.split('camera.png')[0] + "frame_data.json"
f = open(ann_path)
data = json.load(f)
all_boxes = []


proj = numpy.array([
        [0.814897656,0.0,0.0],
        [0.0,1.73205078,0.0],
        [0.0,0.0,-1.0006001]
       ])

count = 0
for ann in data['captures'][0]['annotations'][0]['values']:
    trans = ann['translation']
    rot = ann['rotation']
    #rotation = Quaternion(rot[3],rot[2],rot[1],rot[0])
    rotation = Quaternion(rot[0],rot[1],rot[2],rot[3])
    size = ann['size']
    label = ann['labelName']
    bbox = BBox3D(trans,size,label,count,1,rotation)
    all_boxes.append(bbox)
    count = count + 1

count = 0 
out = '/data/out/'
for box in all_boxes:
    est = add_single_bbox3d_on_image(img,box,proj)
    cv2.imwrite(out + str(count)+'.png',est)
    count = count + 1

Below I am attaching the annotation json file, input image and output image.

{
  "frame": 3,
  "sequence": 1,
  "step": 0,
  "timestamp": 0.0,
  "captures": [
    {
      "@type": "type.unity.com/unity.solo.RGBCamera",
      "id": "camera",
      "description": "",
      "position": [
        0.0,
        1.0,
        -10.0
      ],
      "rotation": [
        0.0,
        0.0,
        0.0,
        1.0
      ],
      "velocity": [
        0.0,
        0.0,
        0.0
      ],
      "acceleration": [
        0.0,
        0.0,
        0.0
      ],
      "filename": "step0.camera.png",
      "imageFormat": "Png",
      "dimension": [
        1101.0,
        518.0
      ],
      "projection": "Perspective",
      "matrix": [
        0.814897656,
        0.0,
        0.0,
        0.0,
        1.73205078,
        0.0,
        0.0,
        0.0,
        -1.0006001
      ],
      "annotations": [
        {
          "@type": "type.unity.com/unity.solo.BoundingBox3DAnnotation",
          "id": "bounding box 3D",
          "sensorId": "camera",
          "description": "Produces 3D bounding box ground truth data for all visible objects that bear a label defined in this labeler's associated label configuration.",
          "values": [
            {
              "instanceId": 1,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                -0.852356,
                1.54649019,
                7.443745
              ],
              "size": [
                0.312455267,
                2.644148,
                0.105378032
              ],
              "rotation": [
                0.580162466,
                0.762785256,
                -0.0170957763,
                0.28509298
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 2,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                -0.235688314,
                1.74439812,
                4.853638
              ],
              "size": [
                0.499991477,
                4.414835,
                0.139880344
              ],
              "rotation": [
                -0.563666,
                -0.2809302,
                0.233530492,
                0.740825534
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 3,
              "labelId": 1,
              "labelName": "ball",
              "translation": [
                2.00813437,
                1.12175477,
                7.0
              ],
              "size": [
                0.163441733,
                0.214070022,
                0.244972438
              ],
              "rotation": [
                0.355775416,
                -0.6403569,
                0.6365056,
                0.241303921
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 4,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                -0.715723634,
                1.24168968,
                7.096243
              ],
              "size": [
                0.15947777,
                4.22101641,
                0.1236633
              ],
              "rotation": [
                -0.0178004019,
                0.08918417,
                0.214659438,
                -0.9724457
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 8,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                1.7423532,
                -1.27903175,
                6.8843255
              ],
              "size": [
                0.240564927,
                2.98338675,
                0.225769565
              ],
              "rotation": [
                0.6551384,
                0.385196269,
                0.527974665,
                -0.379025519
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 9,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                -0.191641748,
                -2.87142277,
                7.51928
              ],
              "size": [
                0.5629913,
                2.49054527,
                0.2928295
              ],
              "rotation": [
                0.3242095,
                0.567037344,
                0.6244339,
                -0.428298
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 10,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                -0.5724124,
                -1.73713434,
                7.52793169
              ],
              "size": [
                0.42610836,
                1.77855992,
                0.266619742
              ],
              "rotation": [
                -0.0989798754,
                -0.518722236,
                -0.504769862,
                -0.682889163
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 11,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                1.57338989,
                -3.16453338,
                7.79668856
              ],
              "size": [
                0.198190346,
                2.11251521,
                0.118497737
              ],
              "rotation": [
                0.85455966,
                -0.175006866,
                -0.3341995,
                0.3569469
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            },
            {
              "instanceId": 12,
              "labelId": 2,
              "labelName": "bat",
              "translation": [
                1.99143839,
                0.299092025,
                7.845733
              ],
              "size": [
                0.324493468,
                2.21375871,
                0.277660578
              ],
              "rotation": [
                0.415438116,
                0.46729672,
                0.16943793,
                0.7617977
              ],
              "velocity": [
                0.0,
                0.0,
                0.0
              ],
              "acceleration": [
                0.0,
                0.0,
                0.0
              ]
            }
          ]
        }
      ]
    }
  ],
  "metrics": [
    {
      "@type": "type.unity.com/unity.solo.GenericMetric",
      "id": "scenario_iteration",
      "sensorId": "",
      "annotationId": "",
      "description": "Iteration information for dataset sequences",
      "value": 1
    },
    {
      "@type": "type.unity.com/unity.solo.GenericMetric",
      "id": "random-seed",
      "sensorId": "",
      "annotationId": "",
      "description": "The random seed used to initialize the random state of the simulation. Only triggered once per simulation.",
      "value": 539662031
    }
  ]
}