Implementation of the 3D Bounding Box Estimation (3.2 part in the paper)

[maudzung]

It should be argmin instead of argmax in the formula (7)

This part can be solved by using the g2opy library.
Welcome to contribute to the implementation of this part!
Thanks!

[DMing11]

I have trained your code on KITTI, but got a bad result when I test it,
Do you have got a good result?
Thanks!

[maudzung]

Hi @DMing11
The results have been not good, but also not bad.
The results for 3D bounding boxes will definitely be optimized by one more step that I mentioned above. Please do it if you can. Many thanks!

[DMing11]

Hi @maudzung
Can you release your pre-trained models?
I want to try part 3.2 in paper , but I don't have a relatively good pre-trained models.
Thanks!

[SPengLiang]

scipy.optimize.minimize can help you with minimizing the reprojection error of keypoints.

[maudzung]

Hi @SPengLiang
Many thanks. Did you implement the part? Can you please provide the source code or contribute to this repo?

[SPengLiang]

#=================================
Here are some of my code in my previous works, and it works well. You can customize it to fit your code, such as corner order and optimizer.
#=================================

from scipy.optimize import minimize
def opt(P2, kpts, pos, dim, Ry):
    init_x, init_y, init_z = pos
    init_h, init_w, init_l = dim
    init_Ry = Ry
    fx, fy = P2[0, 0], P2[1, 1]
    cx, cy = P2[0, 2], P2[1, 2]

    # normalize image plane
    norm_u = (kpts[0] - cx) / fx
    norm_v = (kpts[1] - cy) / fy

    def f_rect(states):  
        x = states[0]
        y = states[1]
        z = states[2]
        Ry = states[3]
        h, w, l = init_h, init_w, init_l
        
        # 8 corners. note that it should align with the keypoints order.
        shift = np.array([[1 / 2, 1 / 2, -1 / 2, -1 / 2, 1 / 2, 1 / 2, -1 / 2, -1 / 2],
                          [0, 0, 0, 0, -1 , -1, -1, -1],
                          [1 / 2, -1 / 2, -1 / 2, 1 / 2, 1 / 2, -1 / 2, -1 / 2, 1 / 2],
                          [1, 1, 1, 1, 1, 1, 1, 1]])

        c = np.cos(Ry)
        s = np.sin(Ry)
        rot_mat =  np.array([[c, 0, s, 0],
                             [0, 1, 0, 0],
                             [-s, 0, c, 0],
                             [0, 0, 0, 1]])

        lhw = np.array([[l], [h], [w], [1]])
        lhw_t = np.tile(lhw, [1, 8])
        xyz_shfit = lhw_t * shift
        xyz_shfit = np.dot(rot_mat, xyz_shfit)
        xyz = np.array([[x], [y], [z], [0]]) + xyz_shfit

        xyz[:2, :] /= np.tile(xyz[2:3, :], [2, 1])

        res_u = np.abs(xyz[0] - norm_u) ** 2
        res_v = np.abs(xyz[1] - norm_v) ** 2

        return np.sum(res_u) + np.sum(res_v)

    res = minimize(f_rect, [init_x, init_y, init_z, init_Ry])

    return res.x, res.success

[maudzung]

Great @SPengLiang
Thank you once again😍

[SPengLiang]

@maudzung You're quite welcome. You probably need to change Ry to -Ry as the orientation I used may be different from KITTI's, and the order of keypoints in the optimize function should align with that of your predicted keypoints.

[sparro12]

How can I go about implementing the code from @SPengLiang to help the 3D bounding boxes perform better? More specifically, where is formula(7) in the repo as of now? Does it exist, yet?