Render nearest training view #2384

AdamRashid96 · 2023-08-30T04:05:18Z

Added logic to render the nearest training view to each camera in the rendered path.
The nearest training image is picked through a weighted sum of positional and rotational distance. Cameras that are occluded from the current rendered view are not selected unless there are no other training cameras unoccluded (for cases if the rendered view is in an object)
Not sure whats the best way to inform people about this. Right now you can select this by adding "nearest_camera" to rendered_output_names

test1.mp4

ethanweber

Left a few comments. I think we could Tweet about this feature at some point, as the way to let people know about it.

ethanweber · 2023-08-30T19:26:59Z

nerfstudio/scripts/render.py

@@ -47,11 +47,19 @@
 from torch import Tensor
 from typing_extensions import Annotated

+from scipy.spatial.transform import Rotation as R


Personally I'm not a fan of using R just in case someone uses that letter somewhere in the file. What do you think?

Agreed I got rid of the as R

ethanweber · 2023-08-30T19:31:04Z

nerfstudio/scripts/render.py

+                    true_max_dist, true_max_idx = None, None
+                    for i in range(len(train_cameras)):
+                        train_cam_pos = train_cameras[i].camera_to_worlds[:, 3].cpu()
+                        # Make sure the line of sight from rendered cam to training cam is not blocked by any object


Is this necessary? I think it would be nice if the result is deterministic, regardless of how the NeRF is. Have you found it to be needed in some cases?

I added a flag check-occlusions so you can toggle whether or not you want the check

ethanweber · 2023-08-30T19:36:22Z

nerfstudio/scripts/render.py

+                            max_dist = dist
+                            max_idx = i
+
+                    if max_idx is None:


I think this will never be done? Because of the lines above if max_dist is None or dist < max_dist:

Actually this will occur in cases where every camera is occluded because of the continue in the if statement above

ethanweber · 2023-08-30T19:39:52Z

nerfstudio/scripts/render.py

+                    img = train_dataset.get_image(max_idx)
+                    img = img.permute(2, 0, 1)
+                    img = transforms.ToPILImage()(img)
+                    resized_image = img.resize(
+                        (int(cameras.image_width[0]), int(cameras.image_height[0])), PIL.Image.Resampling.LANCZOS
+                    )
+                    resized_image = transforms.ToTensor()(resized_image).permute(1, 2, 0)


I think this could go down to two lines and you could avoid the extra imports.

img = train_dataset.get_image(max_idx) img = torch.nn.functional.interpolate( img.permute(2, 0, 1)[None], size=(int(cameras.image_height[0]), int(cameras.image_width[0])) )[0].permute(1, 2, 0)

Yep this looks good. Got rid of the PIL import

tancik · 2023-09-08T00:36:21Z

nerfstudio/scripts/render.py

+        assert pipeline.datamanager.train_dataset is not None
+        train_dataset = pipeline.datamanager.train_dataset
+        train_cameras = train_dataset.cameras.to(pipeline.device)
+


I'm not sure all pipelines will have this. For example, I'm not sure what would happen with the generative pipeines.

I moved this check inside the if render_output_names check so it shouldn't error if other pipelines that don't have train_datasets render.

tancik · 2023-09-08T00:37:30Z

nerfstudio/scripts/render.py

+                max_dist, max_idx = -1, -1
+                true_max_dist, true_max_idx = -1, -1
+
+                if "nearest_camera" in rendered_output_names:


Can we expose this some other way than through render_output_names. It is not discoverable at the moment. Maybe a render flag --render-nearest-camera?

Sounds good, I made it into a flag

kerrj · 2023-09-19T16:40:31Z

nerfstudio/scripts/render.py

@@ -47,13 +47,18 @@
 from torch import Tensor
 from typing_extensions import Annotated

+from scipy.spatial.transform import Rotation


for consistency might be nice to use the viser transform library

kerrj · 2023-09-19T16:41:44Z

nerfstudio/scripts/render.py

@@ -75,6 +80,8 @@ def _render_trajectory_video(
    image_format: Literal["jpeg", "png"] = "jpeg",
    jpeg_quality: int = 100,
    colormap_options: colormaps.ColormapOptions = colormaps.ColormapOptions(),
+    render_nearest_camera=False,
+    check_occlusions: bool = False,


add to docstring

kerrj · 2023-09-19T16:43:01Z

nerfstudio/scripts/render.py

@@ -121,6 +128,20 @@ def _render_trajectory_video(
    with ExitStack() as stack:
        writer = None

+        train_cameras = Cameras(


this should be inside the if render_nearest_camera statement below

kerrj · 2023-09-19T16:44:38Z

nerfstudio/scripts/render.py

+    render_nearest_camera: bool = False
+    """Whether to render the nearest training camera to the rendered camera."""
+    check_occlusions: bool = False
+    """Whether to check occlusions."""


this is kinda vague: "if true, checks line-of-sight occlusions when computing camera distance and rejects cameras not visible to each other"

kerrj · 2023-09-19T16:51:15Z

nerfstudio/scripts/render.py

+                if render_nearest_camera:
+                    img = train_dataset.get_image(max_idx)
+                    resized_image = torch.nn.functional.interpolate(
+                        img.permute(2, 0, 1)[None], size=(int(cameras.image_height[0]), int(cameras.image_width[0]))


this doesn't work properly if the aspect ratio of the render camera is different than the train camera, it should resize the height to render height, and automatically calculate width based on the aspect ratio

kerrj

nice

michaelrubloff · 2023-11-15T03:15:34Z

Left a few comments. I think we could Tweet about this feature at some point, as the way to let people know about it.

Added an article to help inform people! Let me know if I missed anything or if it needs to be corrected: https://neuralradiancefields.io/nerfstudio-introduces-nearest-camera-feature/

AdamRashid96 · 2023-11-15T06:28:32Z

Left a few comments. I think we could Tweet about this feature at some point, as the way to let people know about it.

Added an article to help inform people! Let me know if I missed anything or if it needs to be corrected: https://neuralradiancefields.io/nerfstudio-introduces-nearest-camera-feature/

This is awesome! One thing is that I changed it from adding nearest_camera to rendered_output_names instead to a flag --render-nearest-camera. If you could update the article that would be great.

michaelrubloff · 2023-11-15T17:14:05Z

Updated!

AdamRashid96 added 4 commits August 1, 2023 17:49

render nearest camera to render view

f144f0e

added lpips

33abff5

added the rotation distance and the depth check for the cameras

ead997c

render nearest training img

1acd57f

AdamRashid96 requested a review from ethanweber August 30, 2023 04:05

formatting

e807986

ethanweber reviewed Aug 30, 2023

View reviewed changes

AdamRashid96 added 8 commits September 4, 2023 04:24

add flag for occlusions

665162d

remove commented import

42d230c

remove comments

380f37e

fix pyright test

0da83c8

remove None

261af5c

add assert

d10250b

fix pyright tests

bbc8668

format

b24bf11

AdamRashid96 requested review from tancik and kerrj September 8, 2023 00:53

tancik reviewed Sep 11, 2023

View reviewed changes

AdamRashid96 added 2 commits September 11, 2023 19:41

made it a flag render-nearest-camera

4185199

added default cams and dataset for tests

2fb2bdd

kerrj reviewed Sep 19, 2023

View reviewed changes

kerrj reviewed Sep 25, 2023

View reviewed changes

maintain resolution of training images

3b420fa

kerrj approved these changes Nov 13, 2023

View reviewed changes

Merge branch 'main' into adam/render_training

22f04b0

AdamRashid96 merged commit 5f5562b into main Nov 13, 2023
4 checks passed

AdamRashid96 deleted the adam/render_training branch November 13, 2023 18:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Render nearest training view #2384

Render nearest training view #2384

AdamRashid96 commented Aug 30, 2023

ethanweber left a comment

ethanweber Aug 30, 2023

AdamRashid96 Sep 4, 2023

ethanweber Aug 30, 2023

AdamRashid96 Sep 4, 2023

ethanweber Aug 30, 2023

AdamRashid96 Sep 4, 2023

ethanweber Aug 30, 2023

AdamRashid96 Sep 4, 2023

tancik Sep 8, 2023

AdamRashid96 Sep 11, 2023

tancik Sep 8, 2023

AdamRashid96 Sep 11, 2023

kerrj Sep 19, 2023

kerrj Sep 19, 2023

kerrj Sep 19, 2023

kerrj Sep 19, 2023

kerrj Sep 19, 2023

kerrj left a comment

michaelrubloff commented Nov 15, 2023

AdamRashid96 commented Nov 15, 2023

michaelrubloff commented Nov 15, 2023

Render nearest training view #2384

Render nearest training view #2384

Conversation

AdamRashid96 commented Aug 30, 2023

ethanweber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kerrj left a comment

Choose a reason for hiding this comment

michaelrubloff commented Nov 15, 2023

AdamRashid96 commented Nov 15, 2023

michaelrubloff commented Nov 15, 2023