Add fisheye projection support for skybox#8577
Conversation
Extends fisheye projection to the skybox so sky distortion matches Gaussian splat distortion when using wide-angle/fisheye rendering. Made-with: Cursor
There was a problem hiding this comment.
Pull request overview
Extends fisheye projection support to skybox rendering so environment distortion matches Gaussian splat fisheye output, including multi-camera support and example updates.
Changes:
- Add
Sky.fisheye(0–1) and per-camera prerender uniform updates; addSky.destroy()and invoke it fromScene.destroy(). - Implement inverse fisheye mapping in skybox fragment shaders (GLSL + WGSL) and adjust vertex path to ensure full screen coverage at extreme FOVs.
- Fix
FisheyeProjectionto compute independent X/Y scales; updatelod-streamingexample to sync sky fisheye and add HDR CameraFrame toggle + more environments.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/scene/skybox/sky.js | Adds sky fisheye property, prerender uniform updates, and cleanup via destroy() |
| src/scene/shader-lib/wgsl/chunks/skybox/vert/skybox.js | Adds fisheye-specific clip-space varying and fixed-projection coverage path |
| src/scene/shader-lib/wgsl/chunks/skybox/frag/skybox.js | Adds inverse fisheye mapping path to reconstruct view direction in fragment |
| src/scene/shader-lib/glsl/chunks/skybox/vert/skybox.js | GLSL equivalent of fisheye varying + fixed-projection coverage logic |
| src/scene/shader-lib/glsl/chunks/skybox/frag/skybox.js | GLSL equivalent inverse fisheye mapping path |
| src/scene/scene.js | Switches scene destruction to call this._sky.destroy() for listener cleanup |
| src/scene/gsplat-unified/fisheye-projection.js | Computes fisheye projection scales independently for X/Y to avoid singularities |
| examples/src/examples/gaussian-splatting/lod-streaming.example.mjs | Syncs sky fisheye with slider; adds CameraFrame HDR toggle; updates env presets |
| examples/src/examples/gaussian-splatting/lod-streaming.controls.mjs | Adds UI controls for new environments and CameraFrame toggle |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Projection-dependent values derived from the camera's projection matrix. | ||
| // Compute X and Y scales independently to avoid the 0/0 singularity at 180° FOV | ||
| // and to correctly handle the non-linear fisheye mapping for non-square aspect ratios. | ||
| const maxTheta = Math.min(k * Math.PI / 2, 3.13); | ||
| const effHalfFov = Math.min(halfFovX, maxTheta - 0.01); | ||
| const gFov = k * Math.tan(effHalfFov / k); | ||
| const pm00 = this.cornerScale / gFov; | ||
| const cs = this.cornerScale; | ||
|
|
||
| const halfFovX = Math.atan2(1.0, p00); | ||
| const effHalfFovX = Math.min(halfFovX, maxTheta - 0.01); | ||
| this.projMat00 = cs / (k * Math.tan(effHalfFovX / k)); | ||
|
|
||
| const halfFovY = Math.atan2(1.0, p11); | ||
| const effHalfFovY = Math.min(halfFovY, maxTheta - 0.01); | ||
| this.projMat11 = cs / (k * Math.tan(effHalfFovY / k)); | ||
|
|
There was a problem hiding this comment.
FisheyeProjection.update() can force t = 0 for orthographic projection, and also returns early for t <= 0, but projMat00/projMat11 are not reset in that path. If a consumer still uses these fields (e.g. skybox when SKY_FISHEYE stays defined), they can remain stale from a previous perspective update and produce incorrect mapping when switching cameras/projections. Consider setting projMat00/projMat11 to safe rectilinear values (for perspective: p00/p11) when disabling, or explicitly exposing/using enabled to avoid consuming stale scales.
| const material = this.skyMesh.meshInstance.material; | ||
| material.setParameter('fisheye_k', proj.k); | ||
| material.setParameter('fisheye_invK', proj.invK); | ||
| material.setParameter('fisheye_projMat00', proj.projMat00); | ||
| material.setParameter('fisheye_projMat11', proj.projMat11); |
There was a problem hiding this comment.
FisheyeProjection.update() can disable itself for orthographic cameras (sets t=0), but this code still keeps the SKY_FISHEYE variant active and always pushes proj.k/invK/projMat00/projMat11 into the material. In multi-camera setups with mixed projection types, this can cause the skybox fisheye shader path to run for an orthographic camera using stale projMat00/11 from a previous perspective camera. Consider explicitly handling orthographic cameras here (e.g. disable SKY_FISHEYE for that render, or add a uniform/branch in the fisheye shader to fall back to vViewDir).
| const material = this.skyMesh.meshInstance.material; | |
| material.setParameter('fisheye_k', proj.k); | |
| material.setParameter('fisheye_invK', proj.invK); | |
| material.setParameter('fisheye_projMat00', proj.projMat00); | |
| material.setParameter('fisheye_projMat11', proj.projMat11); | |
| const enabled = proj.t > 0; | |
| this._setFisheyeDefine(enabled); | |
| if (enabled) { | |
| const material = this.skyMesh.meshInstance.material; | |
| material.setParameter('fisheye_k', proj.k); | |
| material.setParameter('fisheye_invK', proj.invK); | |
| material.setParameter('fisheye_projMat00', proj.projMat00); | |
| material.setParameter('fisheye_projMat11', proj.projMat11); | |
| } |
| updateSkyMesh() { | ||
| const texture = this.scene._getSkyboxTex(); | ||
| if (texture) { | ||
| this.resetSkyMesh(); | ||
| this.skyMesh = new SkyMesh(this.device, this.scene, this.node, texture, this.type); | ||
| this.skyMesh.depthWrite = this._depthWrite; | ||
|
|
||
| if (this._fisheye > 0) { | ||
| this._setFisheyeDefine(true); | ||
| } |
There was a problem hiding this comment.
The docs state sky fisheye is only supported for SKYTYPE_INFINITE, but SKY_FISHEYE is enabled whenever this._fisheye > 0 regardless of sky type. For SKYTYPE_BOX/SKYTYPE_DOME this define is effectively a no-op due to !defined(SKYMESH) guards in the shaders, but it still forces compilation of extra shader variants. Consider gating _setFisheyeDefine(true) on this.type === SKYTYPE_INFINITE (and similarly when toggling in the setter).
Avoids compiling unused fisheye shader variants for box/dome sky types. Made-with: Cursor
Extends the fisheye projection support (introduced in #8576 for Gaussian splats) to the skybox, so that the sky distortion matches the splat distortion when using wide-angle/fisheye rendering.
Changes:
Sky.fisheyeproperty that controls fisheye projection strength forSKYTYPE_INFINITEskyboxesSKY_FISHEYEdefinexywas a varying to correctly reconstruct NDC in the fragment shader, avoiding perspective-correct interpolation artifactsScene.EVENT_PRERENDERper-camera event to update fisheye uniforms on the sky material, supporting multi-camera setupsSky.destroy()method for proper event listener cleanup, called fromScene.destroy()FisheyeProjectionto compute X and Y projection scales independently, avoiding a singularity at 180° FOVlod-streamingexample: syncsky.fisheyewith the fisheye slider, add CameraFrame toggle for HDR rendering, add more Poly Haven HDRI environmentsAPI Changes:
Sky.fisheyeproperty (number, 0–1): controls fisheye distortion strength for infinite skyboxesSky.destroy()methodExamples:
lod-streamingexample with sky fisheye support, CameraFrame toggle, and additional HDRI environments