Replay renderer: add line-rendering and unproject() #2057

eundersander · 2023-04-03T14:31:45Z

Motivation and Context

Adding some features (and hacks) to our experimental replay renderers, needed for a Python application for SIRo.

integrate DebugLineRender into replay renderers
add ClassicReplayRenderer.unproject
stub implementation for BatchReplayRenderer.unproject
lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly

How Has This Been Tested

Local testing.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes.
All new and existing tests passed.

eundersander · 2023-04-03T14:44:50Z

src/esp/bindings/SimBindings.cpp

+          R"(Retrieve the depth buffer as a CUDA device pointer.)")
+      .def("debug_line_render", &AbstractReplayRenderer::getDebugLineRender,
+           R"(Get visualization helper for rendering lines.)")
+      .def("unproject", &AbstractReplayRenderer::unproject,


Some context: I'm writing a SIRo Python application that uses this. That app sets up a division between the renderer and the sim, paving the way for these pieces to live in separate processses or separate machines across a network (client/server architecture). Eventually, we need a way for sim code to unproject from screen space to world space, e.g. given a mouse cursor position (or arbitrary screen position), compute a world ray. I'd eventually like a replay renderer to provide an unprojectHelper object that encapsulates the camera matrix (including projection/viewport params). This object can be serialized and passed back to the sim, where the sim can do as many arbitrary unprojects as it wants. For now, as a bandaid, I'm having the renderer unproject the mouse cursor position, call this helper, and return the world ray to the sim.

See the unproject usage here (WIP):
https://github.com/eundersander/habitat-lab/blob/eundersander/magnum_glfw_interactive_play/habitat-lab/habitat/utils/gui_app_wrapper.py#L189

TLDR: this API should eventually go away in favor of a getUnprojectHelper API.

eundersander · 2023-04-03T14:47:09Z

src/esp/bindings/SimBindings.cpp

@@ -454,7 +454,11 @@ void initSimBindings(py::module& m) {
          [](AbstractReplayRenderer& self) {
            return py::capsule(self.getCudaColorBufferDevicePointer());
          },
-          R"(Retrieve the depth buffer as a CUDA device pointer.)");
+          R"(Retrieve the depth buffer as a CUDA device pointer.)")
+      .def("debug_line_render", &AbstractReplayRenderer::getDebugLineRender,


Context: see my earlier comment about separate sim and renderer. We eventually need a way for the sim to convey lines to the renderer. For now in my SIRo Python app, since both pieces live in the same thread, I'm simply giving the sim direct access to the debug_line_render object where it can directly add lines. See here:
https://github.com/eundersander/habitat-lab/blob/eundersander/magnum_glfw_interactive_play/habitat-lab/habitat/utils/gui_app_wrapper.py#L219

eundersander · 2023-04-03T14:50:07Z

src/esp/sim/BatchReplayRenderer.cpp

@@ -256,6 +257,9 @@ void BatchReplayRenderer::doRender(
                 "with a standalone renderer", );
  static_cast<gfx_batch::RendererStandalone&>(*renderer_).draw();

+  // todo: integrate debugLineRender_->flushLines
+  CORRADE_INTERNAL_ASSERT(!debugLineRender_);


I'm not using this version of doRender in my SIRo Python app. Taking a shortcut here because I'm rushing to unblock some SIRo team members this week. A point of tension right now is that the SIRo project is using habitat-sim main, not a branch.

On my "eventually nice to have" TODO list I have a batch-friendly line generator for the new line shader (right now it's only possible to generate & draw a line mesh as a whole, not gradually add to it).

Depending on how soon you'd like to have it working for the batch renderer, I can prioritize it.

Related FYI, I can't think of a use case where we'll need line-rendering for a multi-tile (multi-env) setup. I.e. thick lines are only for showing to humans, not AI model training, because they are fundamentally a UX thing and not photorealistic. They're only for single-tile usage of the batch renderer.

But to be clear, we are prioritizing using the batch renderer for a Magnum-based web/VR render client this year ("Habitat Online"), where we will need lines. Just wanted to be clear that it will be single-tile usage of the batch renderer.

eundersander · 2023-04-03T14:51:40Z

src/esp/sim/BatchReplayRenderer.cpp

+    const Mn::Vector2i& viewportPosition) {
+  // temp stub implementation: produce a placeholder ray that varies with
+  // viewportPosition
+  return esp::geo::Ray(


Another shortcut here. We can implement this properly later. As is, this will be useful for helping @0mdc integrate the batch replay renderer into the SIRo Python app.

eundersander · 2023-04-03T14:52:14Z

src/esp/sim/ClassicReplayRenderer.cpp

@@ -31,6 +31,10 @@ ClassicReplayRenderer::ClassicReplayRenderer(
  resourceManager_ =
      std::make_unique<assets::ResourceManager>(metadataMediator, flags);

+  // hack to get ReplicCAD non-baked stages to render correctly
+  resourceManager_->getShaderManager().setFallback(


Long-term, I hope @0mdc can figure out why the lighting isn't working correctly for the classic replay renderer.

I remember I was doing some material patching for the ReplicaCAD composites, in particular adding or removing the flat/unlit material bit (not sure which was the case exactly), because otherwise both the prebaked and non-baked arch files were rendered with the same shader, looking wrong.

So if this works for you for the batch renderer & my composites but not the classic one, that might be the reason.

* add ClassicReplayRenderer.unproject * stub implementation for BatchReplayRenderer.unproject * lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly

0mdc

LGTM!

src/esp/sim/AbstractReplayRenderer.h

* skip unsupported 3D primitives (#2054) * bugfix for setArticulatedObjectModelFilename causing garbage log output (#2053) * Gfx-replay polish: workaround for material-overrides; new keyframe getter (#2035) * gfx-replay polish: workaround for material-overrides; new keyframe getter API needed for a python application * --Have pre-commit use node version compatible with Ubuntu 18.04 (#2058) * --use system node install if present * --change to hook-specific node version specification * --change to nearest official release version * Fix viewer.py framebuffer size mismatch. (#2055) * Fix viewer.py framebuffer size mismatch. * Change int tuples to mn.Vector2. * Minor simplifications. * Replay renderer: add line-rendering and unproject() (#2057) * integrate DebugLineRender into replay renderers * add ClassicReplayRenderer.unproject * stub implementation for BatchReplayRenderer.unproject * lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly * bugfix: cylinder proimitive collsion shape did not use halflength (#2060) * --have disabled renderer test only execute 1 time (#2065) * --(Bugfix) Fix normal transformation calc and address backface culling in case of negative scaling/reflections (#2062) * --don't use co-factor matrix for transforming pre-calculated normals * --address backface winding in Generic and Phong drawables TODO : Still need to set this up in batch renderer. * --revert change in batch renderer The normal matrix derivation will be changed in magnum to be appropriate, so we will keep the original verbiage for clarity. We keep the changes in the default and PBR renderer calls since we need the determinant to determine if winding order must be reversed for backface culling. * --test negative scaling along 3 axis * --make copy of observation buffer for ground truth observation The buffer is shared, so without a copy subsequent observations would overwrite the gt observation buffer. * Update Magnum submodules to latest. (#2066) * Update magnum submodules. (#2073) * Add runtime perf stats for troubleshooting perf problems (#2070) * add ResourceManager::getDrawableCountAndNumVertices, Sim.get_runtime_perf_stat_values, and related helpers * --Refactor Semantic Mesh loading/flattening to remove deprecated functionality and improve efficiency (#2079) * --refactor to remove deprecated functionality * --reviewer suggestions; * --fixed inappropriate alloc * Articulated object skinning (#2076) * Add rig property to RenderAssetInstanceCreationInfo. * Add render_asset to ao_config.json. * Create render asset from ao_config.json when creating an articulated object. * Store bone names into MeshTransformNode. * Add skinning asset loading and phong rendering. * Assorted minor fixes. * Add flag to render articulated object primitives while having a skinned mesh for debugging. * Add skinned articulated object test. * Fix MetadataMediatorTest * Cache joint transformations in drawables, other minor fixes. * Fix test asset path + other minor fixes. * Fix skinned mesh caching. Make ao_config.json render_asset path relative to the file. Code clean-up. * --Add access to Scene Instance-level user defined attributes. (#2081) * --add access to user defined in scene instance configs. * --If SceneDataset or SceneInstance do not exist, return nullptr. --------- Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com> * --Don't make needless copy of scene instance attributes when retrieving user defined values (#2082) * --don't make needless copy of scene instance attributes * --add a test * gfx-replay: fix to remove reflection when converting node transform matrix to rotation/translation (#2085) * Articulated object semantics (#2086) * Parse semantic_id from ao_config.json. * Propagate semantic_id to skinned mesh. * Add skinned mesh semantics test. * Propagate semantics for non-skinned articulated object. * --Convert materials to use magnum materials. (#2083) * --add Mn::MaterialTools; WIP * --address test issue; * --convert materials to magnum materials * --address fallback material not having defaults * --fix many attribute accessor bugs. * --appropriately make new attribute name with lowercase letter * --Flat/Phong and Semantic textures work. Have to fix building PBR layers * --improve ptr attribute naming; organize custom attribute assignment Still need to build texture pointer attribute array with layers if they exist in base material * --use owning material constructors * --cleanup; clang-tidy; fix flat vs phong ambient txtr map in phong shader Flat now has ambient Texture mapped directly from source material, so no need to check baseColorTexture anymore. * --get layers working properly * --support noneRoughnessMetallic texture; handle per-vert objectID for PBR drawables * --cache all material quantities in drawable to speed up draw access * --prepopulate normal texture * --address inappropriate bitflag check; add vertexID flag to PBR shader (#2090) * enable frustum-culling for classic replay renderer (#2096) * enable recompute_navmesh when creating sim with create_renderer==false (#2097) * --[BUGFIX] Reverse directional lights before sending to uniform; (#2094) * --reverse directional lights before sending to uniform; Also correct existing light setups * --fix lighting tutorial lights * --correct orientations of sample and default light with names * Update Magnum submodules. (#2100) * --[BUGFIX] Single channel texture support (#2102) * --Address uncompressed single and double channel textures by modifying swizzles to be rrr1 and rrrg respectively. * add intertiaFromURDF option to URDF loading API (#2098) * add additional magnum importer plugins to python build dependencies * refactor to reduce redundancy --------- Co-authored-by: Eric Undersander <eundersander@fb.com> Co-authored-by: John Turner <7strbass@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com> Co-authored-by: Vladimír Vondruš <mosra@centrum.cz>

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Apr 3, 2023

eundersander requested review from 0mdc and mosra April 3, 2023 14:31

eundersander commented Apr 3, 2023

View reviewed changes

eundersander added 2 commits April 3, 2023 10:55

* integrate DebugLineRender into replay renderers

1dbde98

* add ClassicReplayRenderer.unproject * stub implementation for BatchReplayRenderer.unproject * lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly

clang-tidy fixes

004e098

eundersander force-pushed the eundersander/replay_renderer_lines_and_unproject branch from 1b4e422 to 004e098 Compare April 3, 2023 15:10

0mdc approved these changes Apr 3, 2023

View reviewed changes

src/esp/sim/AbstractReplayRenderer.h Show resolved Hide resolved

src/esp/sim/AbstractReplayRenderer.h Show resolved Hide resolved

eundersander mentioned this pull request Apr 3, 2023

SIRo Sandbox tool MVP facebookresearch/habitat-lab#1238

Merged

5 tasks

eundersander added 2 commits April 4, 2023 18:15

add unit test for unproject

8a4058d

document unproject

84a4b5f

eundersander merged commit 1d6ff65 into main Apr 5, 2023
1 check passed

eundersander deleted the eundersander/replay_renderer_lines_and_unproject branch April 5, 2023 14:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replay renderer: add line-rendering and unproject() #2057

Replay renderer: add line-rendering and unproject() #2057

eundersander commented Apr 3, 2023 •

edited

eundersander Apr 3, 2023

eundersander Apr 3, 2023

eundersander Apr 3, 2023

mosra Apr 3, 2023

eundersander Apr 3, 2023

eundersander Apr 3, 2023 •

edited

eundersander Apr 3, 2023

eundersander Apr 3, 2023

mosra Apr 3, 2023

0mdc left a comment

Replay renderer: add line-rendering and unproject() #2057

Replay renderer: add line-rendering and unproject() #2057

Conversation

eundersander commented Apr 3, 2023 • edited

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eundersander Apr 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0mdc left a comment

Choose a reason for hiding this comment

eundersander commented Apr 3, 2023 •

edited

eundersander Apr 3, 2023 •

edited