Move Gaussian splatting autograd and pipeline logic from C++ to Python

[fwilliams]

Summary

• Port all C++ autograd functions (projection, rasterization, SH evaluation) to pure Python torch.autograd.Function implementations in a new _gaussian_autograd.py module, making the differentiable pipeline easier to extend and debug.
• Eliminate the C++ GaussianSplat3d class entirely: the Python GaussianSplat3d class now directly manages torch.Tensor attributes instead of delegating to a C++ _impl object.
• Rewrite deduplicatePixels as a pure Python function using PyTorch ops, removing its C++ binding and implementation.
• Refactor PLY I/O to return a flat tuple of tensors (no intermediate struct), and update the viewer to accept individual tensors directly instead of a C++ object.
• Remove ~7,600 lines of dead C++ code (GaussianSplat3d.{h,cpp}, detail/autograd/*, DeduplicatePixelsTest.cpp, GaussianSplat3dCameraApiTest.cpp).
• Simplify pybind bindings to only expose low-level dispatch functions; all autograd wiring now lives in Python.

Test plan

• All 135 Python unit tests pass (cd tests && pytest unit/test_gaussian_splat_3d.py -v)
• All 21 C++ gtests pass (ctest --test-dir build/.../src/tests)
• CI passes (codestyle, DCO, full test matrix)

[matthewdcong]

Thanks, looks good. Do we want to add a Python test for deduplicate pixels since we've removed the C++ test (and implementation)?

[fwilliams]

Good call — added a TestDeduplicatePixels class with 22 parameterized tests (11 cases x int32/int64 dtypes) covering: empty input, single pixel, all unique, some/all duplicates, multi-batch with and without duplicates, round-trip reconstruction, and jagged tensor offset verification. These mirror the deleted C++ DeduplicatePixelsTest suite.

[Copilot]

Pull request overview

This PR migrates Gaussian splatting autograd and pipeline orchestration from a C++ GaussianSplat3d implementation to Python torch.autograd.Function wrappers over lower-level C++ dispatch kernels, while simplifying viewer and PLY I/O APIs to operate on raw tensors.

Changes:

• Introduces fvdb/_gaussian_autograd.py with Python autograd wrappers for projection, rasterization, and SH evaluation dispatch functions.
• Refactors viewer + bindings to accept individual Gaussian tensors (instead of a C++ GaussianSplat3d object) and updates tests accordingly.
• Refactors Gaussian PLY I/O to return/accept a flat tuple of tensors + metadata, and removes now-dead C++ autograd/test code.

Reviewed changes

Copilot reviewed 30 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
fvdb/_gaussian_autograd.py	New Python autograd wrappers calling pybind-exposed forward/backward dispatch functions.
fvdb/__init__.py	Reimplements gaussian_render_jagged in Python using the new autograd wrappers; re-exports evaluate_spherical_harmonics.
fvdb/__init__.pyi	Updates public stubs (but currently diverges from __init__.py exports/signatures).
fvdb/_fvdb_cpp.pyi	Removes C++ GaussianSplat3d/projected types from stubs; adds raw dispatch + PLY I/O function stubs and viewer tensor-based API.
fvdb/viz/_scene.py	Passes raw Gaussian tensors into visualization views instead of ._impl.
fvdb/viz/_gaussian_splat_3d_view.py	Updates view construction to call viewer binding with individual tensors.
src/python/GaussianSplatBinding.cpp	Replaces high-level C++ class bindings with enums + low-level dispatch functions and PLY I/O bindings.
src/python/ViewerBinding.cpp	Updates viewer binding to accept raw tensors for Gaussian splat views and uses ops camera enums.
src/fvdb/detail/viewer/Viewer.h/.cpp	Changes viewer API to accept raw tensors and updates camera model type alias.
src/fvdb/detail/io/GaussianPlyIO.h/.cpp	Changes PLY load/save to operate on flat tensor tuples + metadata (no GaussianSplat3d object).
src/tests/ViewerTest.cpp	Adjusts viewer test to the new PLY return type + tensor-based viewer APIs.
tests/unit/test_gaussian_splat_3d.py	Adds Python unit tests for the new pure-Python pixel deduplication path.
src/tests/CMakeLists.txt / deleted tests	Removes now-obsolete C++ tests for removed C++ APIs.
src/CMakeLists.txt	Removes deleted C++ autograd sources and GaussianSplat3d.cpp from the build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

backward() asserts that grad_means2d, grad_depths, and grad_conics are non-None. In PyTorch autograd these grads can be None when a particular output isn’t used in the loss, which would make this custom Function crash at runtime. Replace the asserts by treating missing grads as zero tensors (or returning early when all required grads are None) before calling into _C.project_gaussians_analytic_bwd.

[swahtz]

I didn't have a chance to review this PR because this was over the weekend, but here are some notes on things that changed in this port we should address. It's also useful to know we don't have coverage to catch some of these situations and we should get an LLM friend to write us some pytests to make sure we have coverage going forward.

Python autograd backward() asserts on None gradients

This is pointed out by Copilot above, but all 5 Python torch.autograd.Function.backward() methods in _gaussian_autograd.py hard-assert that incoming gradient tensors are not None:

assert grad_means2d is not None
assert grad_depths is not None
assert grad_conics is not None

In PyTorch autograd, these grads can be None when a particular output isn't used in the loss (e.g., if you only backprop through rendered images but not depths). The C++ autograd handled undefined tensors gracefully. The Python version will crash with an AssertionError in these edge cases.

No input shape/contiguity validation in gaussian_render_jagged

The C++ validated 6 tensor shapes and 6 contiguity requirements:

TORCH_CHECK(means.rsizes() == torch::IntArrayRef({ggz, 3}), "means must have shape (ggz, 3)");
TORCH_CHECK(quats.rsizes() == torch::IntArrayRef({ggz, 4}), "quats must have shape (ggz, 4)");
// ... etc for scales, opacities, viewmats, Ks
TORCH_CHECK(means.is_contiguous(), "means must be contiguous");
// ... etc

The Python gaussian_render_jagged in __init__.py has no input validation -- no shape checks, no contiguity checks.

No shape/device/dtype validation in from_tensors

The C++ constructor's checkState() validated:

• means shape (N, 3), quats shape (N, 4), logScales shape (N, 3), logitOpacities shape (N,), sh0 shape (N, 1, D), shN shape (N, K-1, D)
• All tensors on the same device
• All tensors are floating-point
• All tensors have the same scalar type

The Python from_tensors() does none of this -- raw tensors are stored without validation, so errors surface inside CUDA kernels.

sh0 2D-to-3D auto-reshape removed

The C++ constructor handled sh0 being passed as (N, D) by automatically unsqueezing to (N, 1, D):

if (mSh0.dim() == 2) {
    mSh0 = mSh0.unsqueeze(1);
}

The Python from_tensors does not handle this, so passing 2D sh0 will cause downstream failures.

No SH degree validation in jagged path

The C++ checked:

TORCH_CHECK(K >= (actualShDegree + 1) * (actualShDegree + 1),
            "K must be at least (shDegreeToUse + 1)^2");

The Python gaussian_render_jagged does not validate that K (number of SH basis functions) is sufficient for the requested SH degree. This could produce silently incorrect SH evaluation.

No w2c/K shape checks or C > 0 check in _do_projection

The C++ validateCameraProjectionArgs checked worldToCameraMatrices.sizes() == {C, 4, 4}, projectionMatrices.sizes() == {C, 3, 3}, and C > 0. The Python _do_projection only checks contiguity, not shapes or non-emptiness.

from_state_dict validation

The C++ loadStateDict had ~25 TORCH_CHECK_VALUE statements validating: all required keys present, gradient accumulation tensor shapes (N,), devices match, dtypes are correct (int32 for step counts/radii), consistency between gradient norms and step counts. The Python from_state_dict does none of this -- it just calls state_dict.get().

cat() method missing

The C++ had a cat() static method with comprehensive validation for concatenating splat objects. There is no Python equivalent.

Deleted C++ tests not fully replaced

GaussianSplat3dCameraApiTest (5 test cases covering AUTO resolution, validation rejection, empty camera batches, distortion ignored for pinhole/ortho, etc.) -- No Python equivalents were added for these camera model dispatch and validation scenarios.