There is an inconsistency with world coordinates to screen coordinates in PerspectiveCameras

[orweiser-code]

🐛 Bugs / Unexpected behaviors

three issues:

1. in the transform_points_screen method - instead of compositing all transforms and applying once on the points - you project first to ndc then transform to screen space (a 2N matmul operation instead of a N matmul operation for N points).
2. When accepting K (as opposed to focal length and principal point) with in_ndc=False you take the matrix as is but don't check if it includes the rotational component from world space to screen space and don't deal with it in the get_full_projection_transform method - resulting in a projection to flipped screen coordinates and wrong origin.
3. Point 2 affects the unproject_points method that uses get_full_projection_transform but doesn't reason about this flip when camera is not defined in ndc space

[orweiser-code]

hey guys this it a real bug... can you please comment?

[bottler]

1. I agree that there's an optimization here to be made in the normal case that the camera defines get_projection_transform (i.e. a linear camera). But I wouldn't call this a "real bug".

2. K is the intrinsic transformation. It is not designed to include anything "from world space". This is by design.

[orweiser-code]

1. of course this is not the bug
2. if it is not designed to hold "from world space" information, then there are some pathways that will create incorrect / inconsistent results:

• The api is set so you can initialize the perspective camera type intirinsics via parameters (focal length & principal point) or matrix (K 4x4) and in either ndc space or screen space.
  take for example intializing in screen space with a K matrix. If like you suggest, there is no need to include the 'rotational component' to get from world coords to screen coords then you will get an upside-down transformation when calling for get_full_projection_transform, which will also result in an upside down transform points method.
• same goes for unproject_points - will be upside down
• in the PerspectiveCamera class the method get_ndc_camera_transform also neglects to deal with the inversion when in_ndc() is false (there is a comment there about expecting an orientation for the points - but it is out of logic if you are working with screen space cameras)
• Alsp but more of a minor issue - if you initialize with a K intrinsic matrix the focal_length attribute is neither updated (i.e it stays 1) nor does it have a get_focal_length like the get_principal_point method which stays updated

Anyway my point is that there are the 4 ways to initialize the model and they all need to be valid and consistent under all usages.

[orweiser-code]

on another note - I have recently written a DistortionCamera which inherits from PerspectiveCamera and deals with all the above and the distortion (a radial polynomial factor distortion correction model). I noticed you recently added a fisheye camera object but i am not sure it covers this case (a "regular" perspective camera with a standard lens distortion).
If you think this can interest you guys I can work on making a PR

[orweiser-code]

@bottler wdyt about the inconsistency in PerpectveCameras?

[ayushjain1144]

#1444 this is also caused due to the issue mentioned by @orweiser-code
My code breaks when supplying K directly with in_ndc=False but is fine when directly supplying focal lengths and principal points. Would be great if something can be done to avoid potential time sink in debugging (maybe a warning if not a change in API). Thank you!

[orweiser-code]

@bottler what do you think can be done? passing R,T in pytorch3d world space + focal + principal point + in_ndc=False creates a wrong camera (projection matrix (K) is flipped in x & y)