Render Omni-directional stereo video (360 3D VR) #1986

cvachha · 2023-05-24T09:11:02Z

Adding the omni-directional stereo camera model to render 3D 360 video and images for VR. This includes the camera models for the left and right eyes and additions to the render script to stack the left and right renders for the final render.

I have also updated the documentation on the custom_dataset page to have information on ODS but let me know if there is a better place to put it.

Adding the omni-directional stereo camera model to render 3D 360 video and images for VR. This includes the camera models for the left and right eyes and additions to the render script to stack the left and right renders for the final render. I have also updated the documentation on the custom_dataset page to have information on ODS but let me know if there is a better place to put it.

tancik · 2023-05-24T17:12:22Z

nerfstudio/cameras/cameras.py

+            ods_theta = -torch.pi * ((x - cx) / fx)[0]
+
+            # local axes of ODS camera
+            ods_x_axis = torch.tensor([1, 0, 0]).to("cuda")


Probably won't make much a difference here since the tensor is small, but it is better to construct the tensor on the target device, rather than move it to the device. Also avoid specifying cuda since users may be using cpu.

somethings like:
ods_x_axis = torch.tensor([1, 0, 0], device=c2w.device)

tancik · 2023-05-24T17:14:21Z

nerfstudio/cameras/cameras.py

+            # assign final camera origins
+            c2w[..., :3, 3] = ods_origins_circle
+
+        if CameraType.OMNIDIRECTIONALSTEREO_R.value in cam_types:


This code is very similar to the left eye. Can you consolidate in a helper function.

tancik · 2023-05-24T17:19:41Z

nerfstudio/cameras/cameras.py

        for value in cam_types:
-            if value not in [CameraType.PERSPECTIVE.value, CameraType.FISHEYE.value, CameraType.EQUIRECTANGULAR.value]:
+            if value not in [
+                CameraType.PERSPECTIVE.value,
+                CameraType.FISHEYE.value,
+                CameraType.EQUIRECTANGULAR.value,
+                CameraType.OMNIDIRECTIONALSTEREO_L.value,
+                CameraType.OMNIDIRECTIONALSTEREO_R.value,
+            ]:


Can you refactor this with the above code into,

for cam_type in cam_types: if cam_type == CameraType.PERSPECTIVE.value: ... continue if cam_type == CameraType.FISHEYE.value: ... continue ... raise ValueError(f"Camera type {cam_type} not supported.")

tancik · 2023-05-24T17:26:56Z

nerfstudio/scripts/render.py

@@ -342,6 +344,16 @@ def main(self) -> None:
        crop_data = get_crop_from_json(camera_path)
        camera_path = get_path_from_json(camera_path)

+        if camera_path.camera_type[0] == CameraType.OMNIDIRECTIONALSTEREO_L.value:
+            # temp folder for writing left and right view renders
+            temp_folder_path = os.path.splitext(str(self.output_path))[0] + "_temp"


Avoid os.path, use pathlib instead
temp_folder_path = self.output_path.parent / (self.output_path.stem + "_temp" )

tancik · 2023-05-24T17:28:30Z

nerfstudio/scripts/render.py

+            # temp folder for writing left and right view renders
+            temp_folder_path = os.path.splitext(str(self.output_path))[0] + "_temp"
+            Path(temp_folder_path).mkdir(parents=True, exist_ok=True)
+            left_eye_path = Path(str(temp_folder_path) + "//ods_render_Left.mp4")


This can be cleaner since temp_folder_path is a Path. One reason what this is preferred is that file paths (ie //) may be different between systems (windows vs linux).
left_eye_path = temp_folder_path / "ods_render_Left.mp4"

tancik · 2023-05-24T17:29:03Z

nerfstudio/scripts/render.py

+            # declare paths for left and right renders
+
+            left_eye_path = self.output_path
+            right_eye_path = Path(str(left_eye_path.parent) + "//ods_render_Right.mp4")


left_eye_path = left_eye_path.parent / "ods_render_Left.mp4"

tancik · 2023-05-24T17:33:35Z

nerfstudio/scripts/render.py

+                self.output_path = Path(str(left_eye_path.parent)[:-5])
+                self.output_path.mkdir(parents=True, exist_ok=True)
+                if self.image_format == "png":
+                    ffmpeg_ods_command = f'ffmpeg -y -pattern_type glob -i "{str(left_eye_path).replace(".mp4","")+"//*.png"}"  -pattern_type glob -i "{str(right_eye_path).replace(".mp4","")+"//*.png"}" -filter_complex vstack "{str(self.output_path)+"//%05d.png"}"'


Use pathlib here and below for renaming
{str(left_eye_path.with_suffix(".png"))}

Updates based on suggested code changes including adding a helper function for ODS and improving path creation. I also added a part where if a user tries to render a video with an output path that doesn't have an extension it will automatically add .mp4 to the end. Otherwise this usually results in an error when there is no file extension when rendering a video.

Updated python formatting for render and camera_paths script

tancik · 2023-05-25T16:19:48Z

nerfstudio/cameras/cameras.py

-                CameraType.OMNIDIRECTIONALSTEREO_R.value,
-            ]:
-                raise ValueError(f"Camera type {value} not supported.")
+        for cam_type in cam_types:


Sorry I should have been more specific. The logic above should exist in each if condition, ie:

for cam_type in cam_types: if cam_type == CameraType.PERSPECTIVE.value: mask = (self.camera_type[true_indices] == CameraType.PERSPECTIVE.value).squeeze(-1) # (num_rays) mask = torch.stack([mask, mask, mask], dim=0) directions_stack[..., 0][mask] = torch.masked_select(coord_stack[..., 0], mask).float() directions_stack[..., 1][mask] = torch.masked_select(coord_stack[..., 1], mask).float() directions_stack[..., 2][mask] = -1.0 elif cam_type == CameraType.EQUIRECTANGULAR.value: # Fill in # Add other cases else: raise ValueError(f"Camera type {cam_type} not supported.")

tancik · 2023-05-25T16:25:24Z

nerfstudio/cameras/cameras.py

-            directions_stack[..., 2][mask] = torch.masked_select(-torch.cos(theta) * torch.sin(phi), mask).float()
-
-        if CameraType.OMNIDIRECTIONALSTEREO_L.value in cam_types:
+        def omni_directional_stereo_camera(eye):


Add _ to the beginning since we don't expect it to be used outside this file, add types, and add docstring.

ie,

def _compute_rays_for_omnidirectional_stereo(eye: Literal["left", "right"]) -> Tuple[Float[Tensor "(fill in shape)", Float[Tensor "(fill in shape)"]: """ Compute the rays for an omnidirectional stereo camera Args: eye: Which eye to compute rays for. Returns: A tuple containing the origins and the directions of the rays. """

It doesn't look like you are using self in this function, you could move it outside the class.

Added the type checking and doc string for the ODS camera rays function and reformatted the cam_type ray if statements.

tancik

LGTM

cvachha changed the title ~~Render Omni-directional stereo (360 3D VR)~~ Render Omni-directional stereo video (360 3D VR) May 24, 2023

tancik reviewed May 24, 2023

View reviewed changes

cvachha and others added 4 commits May 24, 2023 21:40

Merge branch 'main' into origin/cyrus/omni-directional-stereo-video

e5eeaf3

Merge branch 'main' into origin/cyrus/omni-directional-stereo-video

026c75c

Updated formatting for render and cam path scripts

6b2df26

Updated python formatting for render and camera_paths script

tancik reviewed May 25, 2023

View reviewed changes

cvachha and others added 3 commits May 25, 2023 17:42

Adding type checking for ODS function and reformatting camera type code

bc97df1

Added the type checking and doc string for the ODS camera rays function and reformatted the cam_type ray if statements.

Merge branch 'main' into origin/cyrus/omni-directional-stereo-video

9317d6e

Merge branch 'main' into origin/cyrus/omni-directional-stereo-video

d2be06e

tancik approved these changes May 26, 2023

View reviewed changes

Merge branch 'main' into origin/cyrus/omni-directional-stereo-video

577bf8c

cvachha merged commit 6993f75 into main May 27, 2023
4 checks passed

cvachha deleted the origin/cyrus/omni-directional-stereo-video branch May 27, 2023 01:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Render Omni-directional stereo video (360 3D VR) #1986

Render Omni-directional stereo video (360 3D VR) #1986

cvachha commented May 24, 2023

tancik May 24, 2023

tancik May 24, 2023

tancik May 24, 2023

tancik May 24, 2023

tancik May 24, 2023 •

edited

tancik May 24, 2023

tancik May 24, 2023 •

edited

tancik May 25, 2023

tancik May 25, 2023

tancik May 25, 2023

tancik left a comment

Render Omni-directional stereo video (360 3D VR) #1986

Render Omni-directional stereo video (360 3D VR) #1986

Conversation

cvachha commented May 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tancik May 24, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tancik May 24, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tancik left a comment

Choose a reason for hiding this comment

tancik May 24, 2023 •

edited

tancik May 24, 2023 •

edited