Question about the 3D points (pose reference)

[FunkyKoki]

Thanks again for your contribution.

I have another question, how did you get the pose reference files, i.e. reference_3d_5_points_trans.npy, reference_3d_68_points_trans.npy, triangles.npy and vertices_trans.npy ?

[vitoralbiero]

The reference points are either 5 or 68 points extracted from the 3D model vertices.
The indices for the 68 points were provided by the 3D model we are using, and the 5 points we detected in a projection of the 3D model and mapped to their indices.
The triangles were also provided by the 3D model, and are the connections been the vertices.
The vertices we are using are the ones provided by the 3D model but flipped so that the face is facing the camera.

Hope this helps.

[FunkyKoki]

Thanks for your answer, but I am still confusing. (I am deeply sorry to reply so late, because I have been playing Red Dead Redemption 2 for nearly a week).

What is the 3D face model you are using, is it publicly available? And how did you do this projection, is there any reference code?

[vitoralbiero]

I really like RDR2, it is a very great game!

The 3D face model used could be any generic 3D face model, as we are only using it to create the ground-truth pose labels, and rendering. You can download one here.

To map detected 2D points to their corresponding 3D points, you will need to get what we call an unprojection matrix. With that, you can easily map any 2D point in the rendered image to it's corresponding 3D point.
Unfortenately, I do not have code that I can share for this.

[FunkyKoki]

Thank you so much for your answer.

[FunkyKoki]

PS. RDR2 is great indeed.