How to render results with weak perspective camera

[kaufManu]

Thanks a lot for this great and easy-to-use repo!

I'm trying to render the results using the weak perspective camera model. My question relates to these issues:

• 3 camera parameters of weak-perspective camera #134
• perspective_projection #241
• To get cam_intrinsics and cam_extrinsics from .npz files #300

However none of these issues gave me the answer I was looking for. I am using the weak perspective camera parameters stored in cam and as suggested in this issue I multiply them with 2. I also pad the image to be square as mentioned here. I then convert the weak perspective camera model to a projection matrix the same way I used to do it for VIBE, which worked well there. However, for the ROMP output I'm still getting a slight misalignment, as you can see in the following screenshot. The light model is what I am rendering and the blue model in the background is the visualization output from ROMP. I think it's because I should somehow account for cam_trans but I don't know how exactly. Can you help me with this?

[Arthur151]

@kaufManu
I think such mis-alignment is due to the different rendering way, a weak-perspective one v.s. a perspective one.
I guess that the deep blue one is rendered with a weak-perspective camera.
While the light one is rendered with a perspective camera.
Even at the same 3D position in the camera space, such difference in rendering camera would lead to a slight misalignment. The reason why VIBE doesn't have such misalignment is that VIBE always renders with a perspective camera.

[kaufManu]

Thanks for your reply. I agree that if I were to render this with a perspective camera, that we should expect some misalignment. However, in the image above, I'm using the weak-perspective camera model, not a perspective camera, so I think in theory it should be able to overlay exactly with what your code produces (which is the dark blue model in the above image).

Like you explained in this post I'm using result['cam'][0][0] as the scale and 2 * result['cam'][0][1:] as the translation in x and y to obtain the above image with a weak-perspective camera model. I think I have to also use cam_trans for a perfect alignment, but I'm not sure exactly how - e.g., do I just add it to the 3D vertices before the projection?

[kaufManu]

The code I'm using to render this image is now online, in case that helps:

https://github.com/eth-ait/aitviewer/blob/main/examples/load_ROMP.py

[tijiang13]

TO @kaufManu ,

1. From what I can tell @Arthur151 actually rendered the image with a perspective camera (intrinsics camera in pyrender) because he was relying on pyrender for rendering, which didn't implement weak perspective cameras. Therefore he just used a roughly estimated focal length (assuming FOV=60) and assumed the camera pose to be identity matrix. So it does make sense if you got different rendering result.

2. The implementation of weak perspective camera in your snippet kind of puzzles me, because it seems

   1. you assumed camera location to be (0, 0, 1) which if I understood correctly should be cam_trans?

   2. also regarding projection matrix from s and tx ty. Because when converting perspective camera (OpenCV) to OpenGL, for the projection matrix P=NDC@K, P[0, 0] = f * 2 / W, and for weak perspective camera, it should satisfy f = s * W. So in analogy, P[0, 0] should be 2 * s? But I am actually not very sure if the analogy applies here because the rotation will need to be adjusted as well, as for weak perspective camera the third row of R should be set to zero.

To @Arthur151

As stated above, it seems that when you evaluate ROMP on AGORA you use a "fake" focal length (from FOV) and estimated the camera pose with that said focal length. Why didn't you just use the same weak perspective camera and cam_trans as is in training phase?

The "fake" focal length and trans should work fine but it can still be different from the ones from training, is there any specific consideration or simply for convenience?

[Arthur151]

@tijiang13 Good question! Glad that you notice this point.
We intend to unify the way of acquiring the 3D translation in a perspective camera space.
For the ones who can obtain the precise camera intrinsics from the images, such perspective way would be helpful to obtain the real 3D translation in camera space. From my view, sticking to the weak-perspective way would not achieve this goal finally.
Best, looking forward to more dicussions.
Yu

[kaufManu]

Thanks for all your replies. With the help of @tijiang13 we were able to figure this out. The code here now overlais perfectly with what ROMP renders, as can be seen from the following image (the yellow outline is what is rendered on top of the ROMP output).

For future reference, here are some comments and clarifications:

• @Arthur151 I took the deep blue model that was visible on the first image from here which I think used Sim3D to render it.
• To the best of my knowledge, VIBE is not using perspective projection, but indeed a weak-perspective camera (cf. here). I can also get perfect alignment if I use a weak-perspective camera and render the VIBE result again, cf. this example, indicating that they indeed use a weak-perspective projection to do their rendering internally.
• Indeed my assumption was wrong that ROMP used a weak perspective projection for rendering. To get the same rendering as ROMP, the following works:
  • Use --renderer pyrender when running ROMP.
  • To project the ROMP result onto the image plane, use the focal length f = max(cols, rows)/2. * 1./np.tan(np.radians(fov/2)) where fov = 60 as explained by @Arthur151 here.
  • Set the rotation part of the extrinsics to the identity and the translation to cam_trans as estimated by ROMP.
• The above leads to perfect alignment as shown in the image. I never managed to get perfect alignment when using the default renderer when running ROMP, which I think is Sim3D.

[Arthur151]

@kaufManu Yes, you are right.
B.T.W., your aitviewer is just amazing!
I would love to include it into the visualization part of simple-romp if you may allow.
I would also recommand it to my colleagues for visualization.

Besides, it would be great if we can also visualize the camera motion. Because you know that there are some works that also consider the camera motion, like SPEC and GLAMR.

Best,
Yu

[kaufManu]

@Arthur151 Thanks for your kind words!
It would be awesome if you included the viewer in your visualizations! We are actively developing it and also welcome community contributions, so please let me know if you have any questions/problems/feature requests via our Github issue tracker.

Yes I agree, we are actually working on integrating camera motion right now. I'll see if I can add an example from SPEC or GLAMR to our repo with the next release.

[kaufManu]

FYI, visualizing camera motions is now possible, cf. GLAMR example here.

[Arthur151]

Thanks! @kaufManu
I am really impressed by the latest (plentied of) new features, especially the visualization with camera and image demo.
I am re-formulating the whole visualization pipeline with your AITViewer. Hope I can update this part in the next work.
Best regards,
Yu

[kaufManu]

@Arthur151 Amazing! Please let me know (probably best via Github issues) if you encounter any problems or if you have feature requests that would make things easier for you!