Skip to content

feat: add fisheye projection for Gaussian splats#8576

Merged
mvaligursky merged 3 commits intomainfrom
mv-fisheye-projection
Apr 3, 2026
Merged

feat: add fisheye projection for Gaussian splats#8576
mvaligursky merged 3 commits intomainfrom
mv-fisheye-projection

Conversation

@mvaligursky
Copy link
Copy Markdown
Contributor

@mvaligursky mvaligursky commented Apr 3, 2026

Adds fisheye projection support for 3D Gaussian splatting, based on the generalized camera model from Optimal Projection for 3D Gaussian Splatting. The projection uses g(θ) = k·tan(θ/k) with a Jacobian-corrected covariance, enabling wide-angle and "little planet" effects.

Changes:

  • Add GSplatParams.fisheye property (0–1 range) to control fisheye projection strength
  • 0 = standard perspective (original fast path), >0 = increasing barrel distortion
  • Both raster (quad) and compute rendering paths are supported
  • Shader variants (with/without fisheye) are lazily compiled and cached for instant switching
  • Cone-based culling replaces frustum culling when fisheye is active (compute path)
  • New FisheyeProjection helper class computes all derived projection values from the normalized slider input

API Changes:

  • GSplatParams.fisheye: new property (number, 0–1, default 0)

Examples:

  • Updated lod-streaming example with Fisheye slider and FOV clamping for invalid states

Performance:

  • Zero overhead when fisheye is disabled (original shader path, no extra uniforms)
  • Fisheye shader precomputes expensive trigonometry on CPU and shares values between shader stages
  • First enable has a one-off shader compilation cost; subsequent toggles are free

Adds a generalized fisheye projection model (g(θ) = k·tan(θ/k)) with
Jacobian-corrected covariance for both raster and compute GSplat
rendering paths. Controlled via GSplatParams.fisheye (0–1 range).

Based on "Optimal Projection for 3D Gaussian Splatting"
(https://arxiv.org/abs/2402.00752).

Made-with: Cursor
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds fisheye projection support to the unified Gaussian splatting pipeline (quad + compute paths), including shader variants, derived projection parameter computation on CPU, and fisheye-aware culling in the compute interval pipeline.

Changes:

  • Add GSplatParams.fisheye (0–1) and introduce a FisheyeProjection helper to derive projection constants (k, proj scales, maxTheta).
  • Add GSPLAT_FISHEYE shader variants for both GLSL/WGSL vertex paths and WGSL compute path (projection + Jacobian-adjusted covariance).
  • Switch compute interval culling to cone-based culling when fisheye is enabled; update the LOD streaming example UI with fisheye + environment controls.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/scene/shader-lib/wgsl/chunks/gsplat/vert/gsplatStructs.js Adds fisheye per-splat shared values to the vertex center struct (WGSL).
src/scene/shader-lib/wgsl/chunks/gsplat/vert/gsplatCorner.js Implements fisheye Jacobian path for covariance projection (WGSL).
src/scene/shader-lib/wgsl/chunks/gsplat/vert/gsplatCenter.js Adds fisheye projection path and fisheye uniforms (WGSL).
src/scene/shader-lib/wgsl/chunks/gsplat/compute-gsplat-local-tile-count.js Extends compute tile-count shader uniforms / call-site to support fisheye (WGSL).
src/scene/shader-lib/wgsl/chunks/gsplat/compute-gsplat-interval-cull.js Adds fisheye cone-culling variant for interval cull (WGSL).
src/scene/shader-lib/wgsl/chunks/gsplat/compute-gsplat-common.js Adds fisheye projection + Jacobian in shared compute covariance routine (WGSL).
src/scene/shader-lib/glsl/chunks/gsplat/vert/gsplatStructs.js Adds fisheye per-splat shared values to the vertex center struct (GLSL).
src/scene/shader-lib/glsl/chunks/gsplat/vert/gsplatCorner.js Implements fisheye Jacobian path for covariance projection (GLSL).
src/scene/shader-lib/glsl/chunks/gsplat/vert/gsplatCenter.js Adds fisheye projection path and fisheye uniforms (GLSL).
src/scene/gsplat-unified/gsplat-renderer.js Adds shared fisheyeProj helper instance to the renderer base.
src/scene/gsplat-unified/gsplat-quad-renderer.js Toggles GSPLAT_FISHEYE define + binds fisheye uniforms for quad path.
src/scene/gsplat-unified/gsplat-params.js Adds new GSplatParams.fisheye API property + docs.
src/scene/gsplat-unified/gsplat-manager.js Updates culling pipeline to feed fisheye parameters and cone-culling state.
src/scene/gsplat-unified/gsplat-local-dispatch-set.js Adds lazy caching for standard vs fisheye tile-count compute resources.
src/scene/gsplat-unified/gsplat-interval-compaction.js Adds cached compute variants for frustum vs fisheye cone culling + parameter wiring.
src/scene/gsplat-unified/gsplat-frustum-culler.js Stores camera cone-culling inputs (pos/forward/maxTheta) for fisheye mode.
src/scene/gsplat-unified/gsplat-compute-local-renderer.js Selects/creates fisheye vs standard tile-count compute variant and binds uniforms.
src/scene/gsplat-unified/fisheye-projection.js New helper computing derived fisheye projection constants from slider + camera state.
examples/src/examples/gaussian-splatting/lod-streaming.example.mjs Adds fisheye UI wiring, FOV clamping, and HDRI environment presets/loader.
examples/src/examples/gaussian-splatting/lod-streaming.controls.mjs Adds fisheye slider + environment selector; expands FOV slider range.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/scene/shader-lib/wgsl/chunks/gsplat/compute-gsplat-common.js
Comment thread src/scene/gsplat-unified/gsplat-interval-compaction.js Outdated
Comment thread src/scene/gsplat-unified/gsplat-manager.js
Comment thread src/scene/gsplat-unified/gsplat-quad-renderer.js
Comment thread src/scene/gsplat-unified/gsplat-compute-local-renderer.js
Comment thread src/scene/gsplat-unified/fisheye-projection.js Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: graphics Graphics related issue enhancement Request for a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants