Add support for Project Aria datasets. #2617
Conversation
1) An export script for processing Project Aria datasets 2) Add support for Fisheye624 cameras 3) Add support for center-crop masking and sampling Example commands for running nerfstudio on Project Aria data: python scripts/datasets/process_project_aria.py --vrs_file=<path_to_vrs> --mps_data_dir=<path to MPS directory> --output_dir=<nerfstudio_data_dir> ns-train nerfacto --data <nerfstudio_data_dir> nerfstudio-data --orientation-method none
|
Had to fiddle with the From Aria captures in Berkeley:
|
|
Awesome! @brentyi could you expand a bit on what changes you needed to make in your transforms.json paths? Is it something I can update in the code? |
|
|
||
|
|
||
| # unproject 2D point to 3D with fisheye624 model | ||
| def fisheye624_unproject(coords: torch.Tensor, distortion_params: torch.Tensor) -> torch.Tensor: |
There was a problem hiding this comment.
It would nice to annotate (or comment, or assert) the shapes here
|
I'm hesitant to write custom kernels for each camera model, especially
because the affine approximation for Gaussians doesn't hold as well with
more distortion. In the meantime we can undistort the full images in the
gsplat PR.
…On Tue, Nov 21, 2023 at 7:06 PM Brent Yi ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In nerfstudio/scripts/datasets/process_project_aria.py
<#2617 (comment)>
:
> + # create an AriaImageFrame for each image in the VRS.
+ print("Creating Aria frames...")
+ aria_frames = [
+ to_aria_image_frame(provider, index, name_to_camera, t_world_devices, self.output_dir)
+ for index in range(0, provider.get_num_data(stream_id))
+ ]
+
+ # create the NerfStudio frames from the AriaImageFrames.
+ print("Creating NerfStudio frames...")
+ RGB_VALID_RADIUS = 707.5
+ nerfstudio_frames = {
+ "camera_model": ARIA_CAMERA_MODEL,
+ "frames": [to_nerfstudio_frame(frame) for frame in aria_frames],
+ "fisheye_crop_radius": RGB_VALID_RADIUS,
+ }
+
It could also be helpful to save the point cloud here, so #2521
<#2521> can use it
for initialization (once it's merged):
# import pandas as pd
# import open3d as o3d
points_path = self.mps_data_dir / "global_points.csv.gz"
if points_path.exists():
print("Found global points, saving to CSV...")
df = pd.read_csv(points_path, compression="gzip")
points = df[["px_world", "py_world", "pz_world"]].values
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(points)
ply_file_path = self.output_dir / "global_points.ply"
o3d.io.write_point_cloud(str(ply_file_path), pcd)
nerfstudio_frames["ply_file_path"] = "global_points.ply"
else:
print("No global points found!")
We would also need to add support for Fisheye624 in the splatting
pipeline... ideally we would integrate the camera model directly into
gsplat/the rasterization kernels, but maybe just undistorting the label
images is fine too? Seems like that's how the OpenCV fisheye model is being
handled right now.
cc @kerrj <https://github.com/kerrj>, @vye16 <https://github.com/vye16>
—
Reply to this email directly, view it on GitHub
<#2617 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLOKW7F4TDYHU546UVNRCTYFVT33AVCNFSM6AAAAAA7O5CO2CVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONBTGQ3TIOBQG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@vye16 Agreed -- it would be a nightmare to write custom kernels for each projection function. It would be nice if you could use jax or functorch to compute the jacobian of the projection functions dynamically (which would allow 3d -> 2d gaussian projection with correct covariance). Undistorting Aria images for now is the right approach. I'd prefer to leave that for a followup PR if possible |
|
Ok, makes sense to me! Undistortion in a separate PR seems fine. 🙂 |
|
Just wondering @sweeneychris, did you have next steps / a timeline in mind for this PR? |
|
I plan to work on this today and tomorrow so its ready to check in! |
|
Left a last question above but things look ready to me! I made some minor tweaks: a docs update, dependencies, and exposed the Aria executable via Also checked the whole pipeline on my machine; here's a render from the data I collected earlier: aria-nerfstudio-render.mp4 |
| print("Creating NerfStudio frames...") | ||
| CANONICAL_RGB_VALID_RADIUS = 707.5 | ||
| CANONICAL_RGB_WIDTH = 1408 | ||
| rgb_valid_radius = CANONICAL_RGB_VALID_RADIUS * (aria_frames[0].camera.width / CANONICAL_RGB_WIDTH) |
There was a problem hiding this comment.
How did you get these numbers? On the project aria github linked here, I see different numbers; Is there a reason why?
There was a problem hiding this comment.
Do you mean the width? This is just from different recording profiles; the RGB camera can be configured to record at either 1408x1408 (2MP) or 2880x2880 (8MP).
There was a problem hiding this comment.
Gotcha! that makes sense why the comment says "(full res during calibration)"
Example commands for running nerfstudio on Project Aria data: