Squashed commit of the following:

commit d9b022b Author: J.Y <132313008+jb-ye@users.noreply.github.com> Date: Fri Apr 19 14:03:55 2024 -0400 Allow alpha transparency training for blender datasets input (nerfstudio-project#3088) Allow alpha transparency training for rgba input Co-authored-by: Justin Kerr <justin.g.kerr@gmail.com> commit eddf2d2 Author: Congrong Xu <50019703+KevinXu02@users.noreply.github.com> Date: Thu Apr 18 17:37:50 2024 -0700 Changes for trainer.py to support the Gradio webui (nerfstudio-project#3046) * changes for trainer to support webui * Update trainer to support webui * format * add a seperated shutdown() function to stop training * typo fix * get rid of _stop_viewer_server() * Update trainer.py * organize import --------- Co-authored-by: Brent Yi <yibrenth@gmail.com> commit 45d8bb7 Author: Justin Kerr <justin.g.kerr@gmail.com> Date: Wed Apr 17 23:54:43 2024 -0700 updates to apply_pca_colormap (nerfstudio-project#3086) * improvements to pca_colormap: allow input pca matrix, optional ignore_zeros arg * typo commit babf577 Author: Patrick Morgan <patrick@patrick-morgan.net> Date: Fri Apr 12 14:23:47 2024 -0700 Fix call to detect ODM reconstruction file (nerfstudio-project#3018) commit a64026f Author: Justin Kerr <justin.g.kerr@gmail.com> Date: Fri Apr 12 12:32:22 2024 -0700 shift principle points by .5 in undistortion (nerfstudio-project#3071) * shift principle points by .5 * fix ordering * add fisheye offset commit c302cc5 Author: Gunjan Chhablani <chhablani.gunjan@gmail.com> Date: Thu Apr 11 16:41:14 2024 -0400 Add pointcloud loading for blender parser (nerfstudio-project#2902) * Add pointcloud loading for blender parser * Fix argument * Fix docstring * Fix metadata in dataparsers * Add changes from review * Avoid changing default behavior * Apply `scale_factor` to loaded points --------- Co-authored-by: Brent Yi <yibrenth@gmail.com> commit 03bdeff Author: hyperzy <50564821+hyperzy@users.noreply.github.com> Date: Thu Apr 11 14:56:37 2024 -0400 Fixed Unexpectedly huge RAM consumption of parallel datamanager (nerfstudio-project#3060) * Reduced the memory usage by caching image after the datamanager process starts commit 4714ae7 Author: J.Y <132313008+jb-ye@users.noreply.github.com> Date: Thu Apr 11 14:37:02 2024 -0400 write eval combined images for diagnostics (nerfstudio-project#3070) commit eba72db Author: Otto Seiskari <otto.seiskari@gmail.com> Date: Thu Apr 11 21:23:17 2024 +0300 Camera pose optimization for Splatfacto (nerfstudio-project#2891) * Add pose optimization to Splatfacto * Disable Splatfacto pose optimization by default * Improve apply_to_camera for Gaussian Splatting pose optimization Do not chain modifications to camera_to_worlds to improve numerical stability and enable L2 rot/trans penalties. * Add separate mean and max rot/trans metrics to camera-opt * Tweak pose optimization hyperparameters Parameters used in the Gaussian Splatting on the Move paper v1 * Unit test fix for new cameara_optimizer training metrics * Adjust splatfacto-big camera pose optimization parameters Same parameters as in normal Splatfacto --------- Co-authored-by: jh-surh <jh.surh@bucketplace.net> commit 2d9bbe5 Author: dberga <dberga@users.noreply.github.com> Date: Thu Apr 11 04:04:33 2024 +0200 fix exporter.py for FullImageDataManager and RandomCamerasDataManager (nerfstudio-project#3065) * fix exporter.py for FullImageDataManager and RandomCamerasDataManager * sort imports * ruff --------- Co-authored-by: Brent Yi <yibrenth@gmail.com> commit c381fc8 Author: Justin Kerr <justin.g.kerr@gmail.com> Date: Wed Apr 10 18:59:20 2024 -0700 Fix custom elements nested under @Property (nerfstudio-project#3066) * patch custom elements bug * Skip `cached_property` * fomrat * comment, type hint --------- Co-authored-by: Brent Yi <yibrenth@gmail.com> commit d1fc2ee Author: Saurav Maheshkar <sauravvmaheshkar@gmail.com> Date: Mon Apr 8 23:46:00 2024 +0100 feat(ci): migrate to `uv` in CI (nerfstudio-project#3038) * feat(ci): migrate to `uv` in CI * feat(ci): drop upgrade strategy * feat(ci): use `uv` in docs workflow * fix(ci/doc): update sphinx build cmd * fix(ci/doc): specify python path * (experimental) simplify * remove venv * Revert line wrap change for doc.yml --------- Co-authored-by: Brent Yi <yibrenth@gmail.com> commit 2b91715 Author: Sahil Jain <sahilja2@andrew.cmu.edu> Date: Mon Apr 8 13:04:20 2024 -0400 Fix config loading for Eyeful Tower dataset (nerfstudio-project#3054) Add test_filenames to transforms.json generated by eyeful_tower.py commit 26804f8 Author: Ling Jing <111032642+Jing1Ling@users.noreply.github.com> Date: Tue Apr 9 00:07:21 2024 +0800 Make colmapDataParser compatible with 360_v2 dataset format (nerfstudio-project#2860) * added an option to colmapdataparser to round up the image size when downscaling * add round mode and update ffmpeg command * [fix] wrong variable order * update format --------- Co-authored-by: Jing <jing1ling@intel.com> commit 911091c Author: Jan-Niklas Dihlmann <j.dihlmann@googlemail.com> Date: Mon Apr 8 16:17:56 2024 +0200 Added SIGNeRF to Nerfstudio methods (nerfstudio-project#3053) * feat: signerf as nerfstudio method * fix: formating issues and indexing
jancoveeden · Apr 23, 2024 · 6958fe0 · 6958fe0
1 parent 48050dc
commit 6958fe0
Show file tree

Hide file tree

Showing 23 changed files with 361 additions and 70 deletions.
diff --git a/.github/workflows/core_code_checks.yml b/.github/workflows/core_code_checks.yml
@@ -25,17 +25,20 @@ jobs:
           key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
       - name: Install dependencies
         run: |
-          pip install --upgrade --upgrade-strategy eager -e .[dev]
+          pip install uv
+          uv pip install --system --upgrade -e .[dev]
       - name: Run license checks
         run: |
           ./nerfstudio/scripts/licensing/license_headers.sh --check
       - name: Check notebook cell metadata
         run: |
           python ./nerfstudio/scripts/docs/add_nb_tags.py --check
       - name: Run Ruff Linter
-        run: ruff check docs/ nerfstudio/ tests/ --output-format=github
+        run: |
+          ruff check docs/ nerfstudio/ tests/ --output-format=github
       - name: Run Ruff Formatter
-        run: ruff format docs/ nerfstudio/ tests/ --diff
+        run: |
+          ruff format docs/ nerfstudio/ tests/ --diff
       - name: Run Pyright
         run: |
           pyright

diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml
@@ -18,7 +18,8 @@ jobs:
           python-version: '3.9'
       - name: Install dependencies
         run: |
-          pip install .[docs]
+          pip install uv
+          uv pip install --system .[docs]
       - name: Sphinx build
         # fail on warnings
         run: |

diff --git a/docs/index.md b/docs/index.md
@@ -148,12 +148,13 @@ This documentation is organized into 3 parts:
 ### Third-party Methods
 
 - [Instruct-NeRF2NeRF](nerfology/methods/in2n.md): Editing 3D Scenes with Instructions
+- [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions
+- [SIGNeRF](nerfology/methods/signerf.md): Controlled Generative Editing of NeRF Scenes
 - [K-Planes](nerfology/methods/kplanes.md): Unified 3D and 4D Radiance Fields
 - [LERF](nerfology/methods/lerf.md): Language Embedded Radiance Fields
 - [Nerfbusters](nerfology/methods/nerfbusters.md): Removing Ghostly Artifacts from Casually Captured NeRFs
 - [NeRFPlayer](nerfology/methods/nerfplayer.md): 4D Radiance Fields by Streaming Feature Channels
 - [Tetra-NeRF](nerfology/methods/tetranerf.md): Representing Neural Radiance Fields Using Tetrahedra
-- [Instruct-GS2GS](nerfology/methods/igs2gs.md): Editing 3DGS Scenes with Instructions
 - [PyNeRF](nerfology/methods/pynerf.md): Pyramidal Neural Radiance Fields
 - [SeaThru-NeRF](nerfology/methods/seathru_nerf.md): Neural Radiance Field for subsea scenes
 - [Zip-NeRF](nerfology/methods/zipnerf.md): Anti-Aliased Grid-Based Neural Radiance Fields
@@ -162,13 +163,14 @@ This documentation is organized into 3 parts:
 
 ## Quicklinks
 
-|                                                            |                        |
-| ---------------------------------------------------------- | ---------------------- |
-| [Github](https://github.com/nerfstudio-project/nerfstudio) | Official Github Repo   |
-| [Discord](https://discord.gg/RyVk6w5WWP)                   | Join Discord Community |
+|                                                            |                             |
+| ---------------------------------------------------------- | --------------------------- |
+| [Github](https://github.com/nerfstudio-project/nerfstudio) | Official Github Repo        |
+| [Discord](https://discord.gg/RyVk6w5WWP)                   | Join Discord Community      |
 | [Feedback Form](https://forms.gle/sqN5phJN7LfQVwnP9)       | Provide Nerfstudio Feedback |
 
 ## Sponsors
+
 Sponsors of this work includes [Luma AI](https://lumalabs.ai/) and the [BAIR commons](https://bcommons.berkeley.edu/home).
 
 |          |          |

diff --git a/docs/nerfology/methods/index.md b/docs/nerfology/methods/index.md
@@ -29,6 +29,8 @@ The following methods are supported in nerfstudio:
     Instant-NGP<instant_ngp.md>
     Splatfacto<splat.md>
     Instruct-NeRF2NeRF<in2n.md>
+    Instruct-GS2GS<igs2gs.md>
+    SIGNeRF<signerf.md>
     K-Planes<kplanes.md>
     LERF<lerf.md>
     Mip-NeRF<mipnerf.md>
@@ -39,7 +41,6 @@ The following methods are supported in nerfstudio:
     Tetra-NeRF<tetranerf.md>
     TensoRF<tensorf.md>
     Generfacto<generfacto.md>
-    Instruct-GS2GS<igs2gs.md>
     PyNeRF<pynerf.md>
     SeaThru-NeRF<seathru_nerf.md>
     Zip-NeRF<zipnerf.md>

diff --git a/docs/nerfology/methods/signerf.md b/docs/nerfology/methods/signerf.md
@@ -0,0 +1,97 @@
+# SIGNeRF
+
+<h3> Scene Integrated Generation for Neural Radiance Fields</h3>
+
+```{button-link} https://signerf.jdihlmann.com/
+:color: primary
+:outline:
+Website & Code
+```
+
+<video id="teaser" muted autoplay playsinline loop controls width="100%">
+    <source id="mp4" src="https://signerf.jdihlmann.com/videos/SIGNeRF_small.mp4" type="video/mp4">
+</video>
+
+**Generatively edits NeRF scenes in a controlled and fast manner.**
+
+SIGNeRF allows for generative 3D scene editing. We present a novel approach to combine [NeRFs](https://www.matthewtancik.com/nerf) as scene representation with the image diffusion model [StableDiffusion](https://github.com/Stability-AI/stablediffusion) to allow fast and controlled 3D generation.
+
+## Installation
+
+Install nerfstudio dependencies. Then run:
+
+```bash
+pip install git+https://github.com/cgtuebingen/SIGNeRF
+```
+
+SIGNeRF requires to use [Stable Diffusion Web UI](https://github.com/AUTOMATIC1111/stable-diffusion-webui). For detailed installation information follow our [installation guide](https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#installation).
+
+## Running SIGNeRF
+
+Details for running SIGNeRF can be found [here](https://github.com/cgtuebingen/SIGNeRF). Once installed, run:
+
+```bash
+ns-train signerf --help
+```
+
+Two variants of SIGNeRF are provided:
+
+| Method             | Description                | Time  | Quality |
+| ------------------ | -------------------------- | ----- | ------- |
+| `signerf`          | Full model, used in paper  | 40min | Best    |
+| `signerf_nerfacto` | Faster model with Nerfacto | 20min | Good    |
+
+For more information on hardware requirements and training times, please refer to the [training section](https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#training-1).
+
+## Interface
+
+<img src="https://github.com/cgtuebingen/SIGNeRF/raw/main/images/interface.png" width="100%" alt="SIGNeRF Interface" />
+
+SIGNeRF fully integrates into the Nerfstudio [viser](https://viser.studio) interface. It allows for easy editing of NeRF scenes with a few simple clicks. The user can select the editing method, the region to edit, and the object to insert. The reference sheet can be previewed and the NeRF is fine-tuned on the edited images. If you are interested in how we fully edit the Nerfstudio interface please find our code [here](https://github.com/cgtuebingen/SIGNeRF).
+
+## Method
+
+### Overview
+
+SIGNeRF is a novel approach for fast and controllable NeRF scene editing and scene-integrated object generation. We introduce a new generative update strategy that ensures 3D consistency across the edited images, without requiring iterative optimization. We find that depth-conditioned diffusion models inherently possess the capability to generate 3D consistent views by requesting a grid of images instead of single views. Based on these insights, we introduce a multi-view reference sheet of modified images. Our method updates an image collection consistently based on the reference sheet and refines the original NeRF with the newly generated image set in one go. By exploiting the depth conditioning mechanism of the image diffusion model, we gain fine control over the spatial location of the edit and enforce shape guidance by a selected region or an external mesh.
+
+For an in-depth visual explanation and our results please watch our [videos](https://www.youtube.com/playlist?list=PL5y23CB9WmildtW3QyMEi3arXg06zB4ex) or read our [paper](https://arxiv.org/abs/2401.01647).
+
+## Pipeline
+
+<video id="pipeline" muted autoplay playsinline loop controls width="100%">
+    <source id="mp4" src="https://signerf.jdihlmann.com/videos/pipeline_compressed.mp4" type="video/mp4">
+</video>
+
+We leverage the strengths of [ControlNet](https://github.com/lllyasviel/ControlNet), a depth condition image diffusion model, to edit an existing NeRF scene. We do so with a few simple steps in a single forward pass:
+
+0. We start with an original NeRF scene and select an editing method / region
+1. For object generation, we place a mesh object into the scene
+2. And control the precise location and shape of the edit
+3. We position reference cameras in the scene
+4. Render the corresponding color, depth, and mask images, and arrange them into image grids
+5. These grids are used to generate the reference sheet with conditioned image diffusion
+6. Generate new edited images consistent with the reference sheet by leveraging an inpainting mask.
+
+- Repeat step (6) for all cameras
+
+7. Finally, the NeRF is fine-tuned on the edited images
+
+### Reference Sheet Generation
+
+<img src="https://arxiv.org/html/2401.01647v2/x4.png" width="100%" alt="Reference Sheet Generation" />
+
+SIGNeRF uses a novel technique called reference sheet generation. We observe that the image
+diffusion model [ControlNet](https://github.com/lllyasviel/ControlNet) can already generate multiview consistent images of a scene without the need for iterative refinement like [Instruct-NeRF2NeRF](https://instruct-nerf2nerf.github.io/). While generating individual views sequentially introduces too much variation to integrate them into a consistent 3D model, arranging them in a grid of images that are processed by ControlNet in one pass significantly
+improves the multi-view consistency. Based on the depth maps rendered from the original NeRF
+scene we employ a depth-conditioned inpainting variant of ControlNet to generate such a reference sheet of the edited scene. A mask specifies the scene region where the generation should occur. This step gives a lot of control to the user. Different appearances can be produced by generating reference sheets with different seeds or prompts. The one sheet finally selected will directly determine the look of the final 3D scene.
+
+If you want to learn more about the method, please read our [paper](https://arxiv.org/abs/2401.01647) or read the breakdown of the method in the [Radiance Fields article](https://radiancefields.com/controlnet-nerfs-signerf/).
+
+## Related Work
+
+SIGNeRF was inspired by the work of [Instruct-NeRF2NeRF](https://instruct-nerf2nerf.github.io/) which is also available in the [Nerfology](https://docs.nerf.studio/nerfology/methods/in2n.html) documentation.
+
+## Authors
+
+SIGNeRF was developed by [Jan-Niklas Dihlmann](https://jdihlmann.com) and [Andreas Engelhardt](https://aengelhardt.com/).
diff --git a/docs/quickstart/data_conventions.md b/docs/quickstart/data_conventions.md
@@ -12,6 +12,10 @@ We use the OpenGL/Blender (and original NeRF) coordinate convention for cameras.
 
 Our world space is oriented such that the up vector is +Z. The XY plane is parallel to the ground plane. In the viewer, you'll notice that red, green, and blue vectors correspond to X, Y, and Z respectively.
 
+### Pixel coordinates
+
+We assume coordinates correspond to the centers of pixels (e.g. generating a ray for pixel (0,0) will shoot the ray through the center of that pixel). This aligns more with graphics conventions, and is distinct from OpenCV where the corners are aligned with the pixel coordinate.
+
 <hr>
 
 ## Dataset format

diff --git a/nerfstudio/cameras/camera_optimizers.py b/nerfstudio/cameras/camera_optimizers.py
@@ -22,6 +22,7 @@
 from dataclasses import dataclass, field
 from typing import Literal, Optional, Type, Union
 
+import numpy
 import torch
 import tyro
 from jaxtyping import Float, Int
@@ -151,15 +152,20 @@ def apply_to_raybundle(self, raybundle: RayBundle) -> None:
             raybundle.origins = raybundle.origins + correction_matrices[:, :3, 3]
             raybundle.directions = torch.bmm(correction_matrices[:, :3, :3], raybundle.directions[..., None]).squeeze()
 
-    def apply_to_camera(self, camera: Cameras) -> None:
-        """Apply the pose correction to the raybundle"""
-        if self.config.mode != "off":
-            assert camera.metadata is not None, "Must provide id of camera in its metadata"
-            assert "cam_idx" in camera.metadata, "Must provide id of camera in its metadata"
-            camera_idx = camera.metadata["cam_idx"]
-            adj = self(torch.tensor([camera_idx], dtype=torch.long, device=camera.device))  # type: ignore
-            adj = torch.cat([adj, torch.Tensor([0, 0, 0, 1])[None, None].to(adj)], dim=1)
-            camera.camera_to_worlds = torch.bmm(camera.camera_to_worlds, adj)
+    def apply_to_camera(self, camera: Cameras) -> torch.Tensor:
+        """Apply the pose correction to the world-to-camera matrix in a Camera object"""
+        if self.config.mode == "off":
+            return camera.camera_to_worlds
+
+        assert camera.metadata is not None, "Must provide id of camera in its metadata"
+        if "cam_idx" not in camera.metadata:
+            # Evalutaion cams?
+            return camera.camera_to_worlds
+
+        camera_idx = camera.metadata["cam_idx"]
+        adj = self(torch.tensor([camera_idx], dtype=torch.long, device=camera.device))  # type: ignore
+        adj = torch.cat([adj, torch.Tensor([0, 0, 0, 1])[None, None].to(adj)], dim=1)
+        return torch.bmm(camera.camera_to_worlds, adj)
 
     def get_loss_dict(self, loss_dict: dict) -> None:
         """Add regularization"""
@@ -176,8 +182,12 @@ def get_correction_matrices(self):
     def get_metrics_dict(self, metrics_dict: dict) -> None:
         """Get camera optimizer metrics"""
         if self.config.mode != "off":
-            metrics_dict["camera_opt_translation"] = self.pose_adjustment[:, :3].norm()
-            metrics_dict["camera_opt_rotation"] = self.pose_adjustment[:, 3:].norm()
+            trans = self.pose_adjustment[:, :3].detach().norm(dim=-1)
+            rot = self.pose_adjustment[:, 3:].detach().norm(dim=-1)
+            metrics_dict["camera_opt_translation_max"] = trans.max()
+            metrics_dict["camera_opt_translation_mean"] = trans.mean()
+            metrics_dict["camera_opt_rotation_mean"] = numpy.rad2deg(rot.mean().cpu())
+            metrics_dict["camera_opt_rotation_max"] = numpy.rad2deg(rot.max().cpu())
 
     def get_param_groups(self, param_groups: dict) -> None:
         """Get camera optimizer parameters"""

diff --git a/nerfstudio/cameras/cameras.py b/nerfstudio/cameras/cameras.py
@@ -984,12 +984,15 @@ def get_intrinsics_matrices(self) -> Float[Tensor, "*num_cameras 3 3"]:
         return K
 
     def rescale_output_resolution(
-        self, scaling_factor: Union[Shaped[Tensor, "*num_cameras"], Shaped[Tensor, "*num_cameras 1"], float, int]
+        self,
+        scaling_factor: Union[Shaped[Tensor, "*num_cameras"], Shaped[Tensor, "*num_cameras 1"], float, int],
+        scale_rounding_mode: str = "floor",
     ) -> None:
         """Rescale the output resolution of the cameras.
 
         Args:
             scaling_factor: Scaling factor to apply to the output resolution.
+            scale_rounding_mode: round down or round up when calculating the scaled image height and width
         """
         if isinstance(scaling_factor, (float, int)):
             scaling_factor = torch.tensor([scaling_factor]).to(self.device).broadcast_to((self.cx.shape))
@@ -1006,5 +1009,14 @@ def rescale_output_resolution(
         self.fy = self.fy * scaling_factor
         self.cx = self.cx * scaling_factor
         self.cy = self.cy * scaling_factor
-        self.height = (self.height * scaling_factor).to(torch.int64)
-        self.width = (self.width * scaling_factor).to(torch.int64)
+        if scale_rounding_mode == "floor":
+            self.height = (self.height * scaling_factor).to(torch.int64)
+            self.width = (self.width * scaling_factor).to(torch.int64)
+        elif scale_rounding_mode == "round":
+            self.height = torch.floor(0.5 + (self.height * scaling_factor)).to(torch.int64)
+            self.width = torch.floor(0.5 + (self.width * scaling_factor)).to(torch.int64)
+        elif scale_rounding_mode == "ceil":
+            self.height = torch.ceil(self.height * scaling_factor).to(torch.int64)
+            self.width = torch.ceil(self.width * scaling_factor).to(torch.int64)
+        else:
+            raise ValueError("Scale rounding mode must be 'floor', 'round' or 'ceil'.")
diff --git a/nerfstudio/configs/external_methods.py b/nerfstudio/configs/external_methods.py
@@ -14,6 +14,7 @@
 
 
 """This file contains the configuration for external methods which are not included in this repository."""
+
 import inspect
 import subprocess
 import sys
@@ -208,6 +209,22 @@ class ExternalMethod:
     )
 )
 
+# SIGNeRF
+external_methods.append(
+    ExternalMethod(
+        """[bold yellow]SIGNeRF[/bold yellow]
+For more information visit: https://docs.nerf.studio/nerfology/methods/signerf.html
+
+To enable SIGNeRF, you must install it first by running:
+  [grey]pip install git+https://github.com/cgtuebingen/SIGNeRF[/grey] and install Stable Diffusion Web UI see [grey]https://github.com/cgtuebingen/SIGNeRF?tab=readme-ov-file#installation[/grey]""",
+        configurations=[
+            ("signerf", "SIGNeRF method (high quality) used in paper"),
+            ("signerf_nerfacto", "SIGNeRF method combined with Nerfacto (faster training less quality)"),
+        ],
+        pip_package="git+https://github.com/cgtuebingen/SIGNeRF",
+    )
+)
+
 
 @dataclass
 class ExternalMethodDummyTrainerConfig:

diff --git a/nerfstudio/configs/method_configs.py b/nerfstudio/configs/method_configs.py
@@ -632,8 +632,10 @@
         },
         "quats": {"optimizer": AdamOptimizerConfig(lr=0.001, eps=1e-15), "scheduler": None},
         "camera_opt": {
-            "optimizer": AdamOptimizerConfig(lr=1e-3, eps=1e-15),
-            "scheduler": ExponentialDecaySchedulerConfig(lr_final=5e-5, max_steps=30000),
+            "optimizer": AdamOptimizerConfig(lr=1e-4, eps=1e-15),
+            "scheduler": ExponentialDecaySchedulerConfig(
+                lr_final=5e-7, max_steps=30000, warmup_steps=1000, lr_pre_warmup=0
+            ),
         },
     },
     viewer=ViewerConfig(num_rays_per_chunk=1 << 15),
@@ -684,8 +686,10 @@
         },
         "quats": {"optimizer": AdamOptimizerConfig(lr=0.001, eps=1e-15), "scheduler": None},
         "camera_opt": {
-            "optimizer": AdamOptimizerConfig(lr=1e-3, eps=1e-15),
-            "scheduler": ExponentialDecaySchedulerConfig(lr_final=5e-5, max_steps=30000),
+            "optimizer": AdamOptimizerConfig(lr=1e-4, eps=1e-15),
+            "scheduler": ExponentialDecaySchedulerConfig(
+                lr_final=5e-7, max_steps=30000, warmup_steps=1000, lr_pre_warmup=0
+            ),
         },
     },
     viewer=ViewerConfig(num_rays_per_chunk=1 << 15),

diff --git a/nerfstudio/data/datamanagers/full_images_datamanager.py b/nerfstudio/data/datamanagers/full_images_datamanager.py
@@ -347,6 +347,10 @@ def _undistort_image(
                 0,
             ]
         )
+        # because OpenCV expects the pixel coord to be top-left, we need to shift the principal point by 0.5
+        # see https://github.com/nerfstudio-project/nerfstudio/issues/3048
+        K[0, 2] = K[0, 2] - 0.5
+        K[1, 2] = K[1, 2] - 0.5
         if np.any(distortion_params):
             newK, roi = cv2.getOptimalNewCameraMatrix(K, distortion_params, (image.shape[1], image.shape[0]), 0)
             image = cv2.undistort(image, K, distortion_params, None, newK)  # type: ignore
@@ -367,9 +371,13 @@ def _undistort_image(
             mask = torch.from_numpy(mask).bool()
             if len(mask.shape) == 2:
                 mask = mask[:, :, None]
+        newK[0, 2] = newK[0, 2] + 0.5
+        newK[1, 2] = newK[1, 2] + 0.5
         K = newK
 
     elif camera.camera_type.item() == CameraType.FISHEYE.value:
+        K[0, 2] = K[0, 2] - 0.5
+        K[1, 2] = K[1, 2] - 0.5
         distortion_params = np.array(
             [distortion_params[0], distortion_params[1], distortion_params[2], distortion_params[3]]
         )
@@ -388,6 +396,8 @@ def _undistort_image(
             mask = torch.from_numpy(mask).bool()
             if len(mask.shape) == 2:
                 mask = mask[:, :, None]
+        newK[0, 2] = newK[0, 2] + 0.5
+        newK[1, 2] = newK[1, 2] + 0.5
         K = newK
     elif camera.camera_type.item() == CameraType.FISHEYE624.value:
         fisheye624_params = torch.cat(

diff --git a/nerfstudio/data/datamanagers/parallel_datamanager.py b/nerfstudio/data/datamanagers/parallel_datamanager.py
@@ -93,10 +93,10 @@ def __init__(
         self.exclude_batch_keys_from_device = self.dataset.exclude_batch_keys_from_device
         self.pixel_sampler = pixel_sampler
         self.ray_generator = RayGenerator(self.dataset.cameras)
-        self.cache_images()
 
     def run(self):
         """Append out queue in parallel with ray bundles and batches."""
+        self.cache_images()
         while True:
             batch = self.pixel_sampler.sample(self.img_data)
             ray_indices = batch["indices"]