Rasterization using 3d gaussians

[fwilliams]

Summary

This PR adds a dense, differentiable rasterize-from-world path for Gaussian splatting that renders images directly from world-space 3D Gaussians (means/quats/log-scales) instead of relying only on 2D projected conics. It supports UT-based projection for OpenCV camera models, integrates into the GaussianSplat3d public API, and includes extensive regression/gradient tests.

Key changes

• New public API

  • GaussianSplat3d.render_images_from_world(...) (Python + C++ bindings)
  • Supports CameraModel + RollingShutterType and packed OpenCV distortion coefficients.

• New kernels + autograd (dense-only)

  • CUDA forward/backward for rasterization from world-space 3D Gaussians.
  • Autograd wrapper enforces correct backward arity (one grad per forward input).

• Camera model consolidation + reuse

  • Shared CameraModel / RollingShutterType enums (C++) and Python IntEnum wrappers in fvdb.enums.
  • Shared OpenCV distortion math and pose interpolation helpers used across UT + rasterize-from-world.

• Masks + backgrounds parity

  • Optional tile masks and per-camera backgrounds with behavior aligned to the classic rasterizer.
  • Deadlock fixes: block-wide early exits for masked/no-intersection edge tiles in both classic and from-world codepaths.

• Docs + naming cleanup

  • Expanded Python docstrings for render_images_from_world.
  • Removed deprecated aliases and aligned naming (no *_3dgs suffix in the public API).

Test plan

• C++
  • ./build.sh ctest
• Python
  • Run from /tmp so Python imports the installed extension:
    • conda run -n fvdb python -m pytest -q --rootdir /path/to/fvdb-core/tests /path/to/fvdb-core/tests

Notes / follow-ups

• Rasterize-from-world is currently CUDA-only and dense-only (no jagged/sparse output).
• CameraModel / RollingShutterType are kept explicit in some dispatch APIs; folding them into RenderSettings could be a follow-up if we want a single transport object everywhere.

[Copilot]

Pull request overview

This pull request adds 3D Gaussian Splatting (3DGS) rasterization capability that enables rendering directly from world-space Gaussians rather than from 2D projections. This allows geometry gradients to flow through the rasterization step, which is particularly useful for non-pinhole camera models with distortion.

Changes:

• Adds new render_images_from_world_3dgs method that performs ray-ellipsoid rasterization directly in world space
• Introduces CameraModel and RollingShutterType enums for flexible camera projection support
• Implements CUDA kernels for forward and backward passes of the 3DGS rasterizer with support for multiple camera models (pinhole, orthographic, and OpenCV distortion variants)

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/fvdb/detail/ops/gsplat/GaussianCameraModels.h	Defines camera model and rolling shutter enums (contains critical duplication bug)
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGS.cuh	Shared utilities for ray generation, distortion, and coordinate transformations
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGSForward.cu	CUDA kernel for forward rasterization pass
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGSBackward.cu	CUDA kernel for backward gradient computation
src/fvdb/detail/autograd/GaussianRasterizeFromWorld3DGS.{h,cpp}	PyTorch autograd wrapper for the rasterization operation
src/fvdb/GaussianSplat3d.{h,cpp}	Main API implementation routing to appropriate projection and rasterization paths
src/python/GaussianSplatBinding.cpp	Python bindings for the new enums and method
fvdb/gaussian_splatting.py	Python wrapper for the new rendering method
fvdb/enums.py	Python enum definitions for camera models and rolling shutter types
fvdb/_fvdb_cpp.pyi, fvdb/init.pyi, fvdb/init.py	Type stubs and exports for the new functionality
tests/unit/test_rasterize_world3dgs.py	Unit tests for gradient correctness and finite difference validation
src/CMakeLists.txt	Build configuration for new source files
src/fvdb/detail/ops/gsplat/GaussianProjectionUT.h	Refactored to use shared enum definitions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[swahtz]

Could we get a description of the PR please?

[fwilliams]

@swahtz : updated:

• I refactored the rasterize-from-world dispatch APIs to take a RenderSettings object (for image size/origin/tileSize), matching the other rasterize utilities.
• I kept CameraModel and RollingShutterType as explicit parameters for now.

Folding camera model + rolling-shutter policy into RenderSettings would be a broader API change since RenderSettings is used across several rasterize/projection utilities that don’t need UT/rolling-shutter camera semantics. Happy to do that as a follow-up if we want to standardize camera transport further.

[harrism]

Thanks for addressing my comments. I'm approving, but I'd like you to generate missing documentation before merging (see 3 comments).

[swahtz]

This is great, thanks for addressing all my notes. In principle, we could refactor all the feature/alpha writing to writeFeature/writeAlpha functions like on the other rasterization pipelines' Args classes but that can be left to further cleanup.