Better Camera unproject #2001

aclegg3 · 2023-02-08T20:14:26Z

Motivation and Context

Previously camera unproject function would return a normalized ray which was useful for distance calculations and raycasting, but could not be used directly with depth image values (on view plane) for depth unprojection into point clouds.

This PR adds two features:

Added normalized variable to unproject (default true)
When normalized==False, unproject now returns a correctly scaled ray direction in world units to get a precise 3D location from a depth pixel value.

How Has This Been Tested

Modified unit test to validate normalized variable.
New unit tests demonstrate depth unprojection and leverage Bullet collision detection to validate the feature.

Example images from optional visual test output showing a sphere object for a random depth unprojected pixel:

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

…led to world coordinates.

…and easier use in different cases

jturner65 · 2023-02-08T20:22:08Z

src/esp/gfx/RenderCamera.cpp

-       ray.origin)
-          .normalized();
+       ray.origin) /
+      1000.0;


Why scaled by 1/1000? Should this be a defined constant somewhere?

Considering the intent of this change, perhaps some documentation regarding rescaling the normalized camera ray to fit the desired use would be more appropriate?

Great question, this is the far plane distance.
I updated the code and docs such that the scaling is on the user side.
Also added a test to validate this.

how about we get the far distance by doing this right there? @mosra does this look appropriate?

auto zFar = Frustum::fromMatrix(projectionMatrix()*cameraMatrix()).far(); auto dist = zFar(3)/zFar.xyz().length();

The Frustum class is unnecessary overhead -- it calculates also all other planes and you need just one of them, and only its distance.

The "most obvious" way to calculate the distance of the far plane is by doing an inverse transform of a point lying on the far plane (i.e., calculating what point gets projected to (0, 0, 1)). Something like this:

projectionMatrix().inverted().transformPoint(Vector3::zAxis(1.0f)).z()

But that's imprecise and not fast either due to the inversion. After looking at the Matrix4::perspectiveProjection() docs and a bit of math suffering I think the equation is like this:

near = projection[3][2]/(projection[2][2] - 1.0f); far = projection[3][2]/(1.0f + projection[2][2]);

I could add those directly on the Matrix4 class I think, just need to do a bit of testing and correctly handle infinite far plane.

Thanks @mosra ! Do we not need the product of the projection & camera transforms for the far plane calculation?

Yes -- the equations I have give you the distance to where far Z plane is relative to the camera. So then if you transform the camera somewhere, you don't need to do the complicated extraction of the far plane again, only transforming that "far vector" with the camera matrix. I think. My math is rusty.

// Negative Z is forward Vector3 cameraRelativeZIsHere = cameraMatrix().transformPoint(Vector3::zAxis(-far));

…far plane paramter

jturner65

LGTM, except for pesky pylint :) .

src/esp/gfx/RenderCamera.cpp

mosra · 2023-02-09T15:28:13Z

src/esp/gfx/RenderCamera.cpp

+
+  if (normalized) {
+    ray.direction = ray.direction.normalized();
+  }


(Again ignore if this utility isn't meant to be used in performance-sensitive scenarios.)

A better way would be to have this split into two functions, one unproject() that doesn't normalize, and an unprojectNormalized() that calls unproject(), normalizes its result and returns it. That way there's no branch. (Though that way it'll be a breaking change, because unproject() will not normalize anymore, so you might want to invent different naming of the two overloads.)

Yeah, thanks. I thought about this also, but didn't want a breaking change or to add more bulk to the API. I think this one can stay for now.

0mdc

LGTM! Left some minor remarks.

0mdc · 2023-02-10T16:13:04Z

src_python/habitat_sim/utils/settings.py

@@ -30,6 +31,7 @@
    "equirect_semantic_sensor": False,
    "seed": 1,
    "physics_config_file": "data/default.physics_config.json",
+    "enable_physics": True,


What are the consequences of enabling physics by default here? Do we want to do this?

Fair point. This means any Simulator initialized from the default settings util will have access to Bullet API (if installed).
Given that tests are passing, I don't think this is problem.

Some justification:

If user installed bullet compatible Habitat build, they should get Bullet API by default.

This incurs very little overhead for static scenes (builds collision meshes, but doesn't do any heavy compute), so it shouldn't reduce performance for non-physics use cases most of the time (e.g. PointNav with HM3D).

This is a fairly new utility Dict for quick-start and is not currently used downstream by Habitat-lab (outside of one test for the rearrange task which is physics based), so it is not likely a breaking change.

We could add the breaking change tag to the PR and mention this in the description.
If you think this warrants more discussion I could move this change to its own PR as it is not necessary here.

Sounds good. I think we can proceed with it here.

@aclegg3 @0mdc What's the verdict on this change? Should we add the breaking change tag? Otherwise, this PR looks good to merge.

I think that it would be preferable to add this change to another PR for traceability. However, I don't feel strongly either way.

0mdc · 2023-02-10T16:18:33Z

tests/test_gfx.py

+                random.randint(0, render_camera.viewport[0] - 1),
+                random.randint(0, render_camera.viewport[1] - 1),
+            )
+            # NOTE: use un-normlized rays scaled to unit z distance for this application


Typo: normlized

aclegg3 added 3 commits February 8, 2023 09:50

unprojected ray directions should not be normalized and should be sca…

f10aaed

…led to world coordinates.

update the test to validate unprojected depth points

21c90f4

add normalization option to unproject API for backward compatibility …

7140c54

…and easier use in different cases

aclegg3 requested review from jturner65, eundersander and 0mdc February 8, 2023 20:14

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Feb 8, 2023

Merge remote-tracking branch 'origin/main' into better-unproject

ce77520

jturner65 reviewed Feb 8, 2023

View reviewed changes

aclegg3 added 2 commits February 8, 2023 13:42

remove assumption of default far plane distance and adjust test with …

70432ba

…far plane paramter

explicitly cast depth obs to ndarray for mypy

476261a

Skylion007 approved these changes Feb 8, 2023

View reviewed changes

aclegg3 added 2 commits February 8, 2023 15:49

fix some bugs

71c31b1

move far plane into C++ function to offload users

c3c66b0

jturner65 approved these changes Feb 9, 2023

View reviewed changes

type cast to fix mypy complaint

fd23b2d

mosra reviewed Feb 9, 2023

View reviewed changes

cache projection matrix inverse

586072c

0mdc approved these changes Feb 10, 2023

View reviewed changes

jturner65 added 2 commits February 21, 2023 14:54

--simplify far distance calc.

d455cd2

--fix typo

189a9f6

jturner65 merged commit 8648a23 into main Feb 21, 2023

jturner65 deleted the better-unproject branch February 21, 2023 23:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better Camera unproject #2001

Better Camera unproject #2001

aclegg3 commented Feb 8, 2023 •

edited by jturner65

jturner65 Feb 8, 2023 •

edited

jturner65 Feb 8, 2023

aclegg3 Feb 8, 2023

jturner65 Feb 8, 2023

mosra Feb 9, 2023

jturner65 Feb 9, 2023

mosra Feb 9, 2023

jturner65 left a comment

mosra Feb 9, 2023

aclegg3 Feb 9, 2023

0mdc left a comment

0mdc Feb 10, 2023

aclegg3 Feb 10, 2023

0mdc Feb 13, 2023

jturner65 Feb 21, 2023

0mdc Feb 21, 2023

0mdc Feb 10, 2023

Better Camera unproject #2001

Better Camera unproject #2001

Conversation

aclegg3 commented Feb 8, 2023 • edited by jturner65

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

jturner65 Feb 8, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jturner65 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0mdc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aclegg3 commented Feb 8, 2023 •

edited by jturner65

jturner65 Feb 8, 2023 •

edited