Hybrid cluster/standard GAS selection by distance

[brendancol]

Context

Follow-up to #74. Roadmap Phase 2 item (see #57). OptiX 9.1 cluster acceleration is supported (see rtx.py cluster accel APIs) and has been validated on A6000. Cluster GAS provides superior BVH quality and ray traversal performance but is more expensive to build. Currently, GAS type is chosen uniformly — either everything is cluster or everything is standard. A hybrid approach uses cluster GAS where it matters (nearby, high-detail geometry) and standard GAS where build cost matters more than traversal quality (distant, simplified geometry).

Goal

Automatically select cluster vs. standard GAS per geometry based on distance from camera, so nearby geometry gets optimal traversal while distant geometry avoids expensive cluster builds.

Current State

• Cluster GAS support is fully operational: PipelineCompileOptions.allowClusteredGeometry = 1, CLAS partitioning, 4-param optixGetTriangleVertexData() (see MEMORY.md)
• Terrain LOD tiles (terrain_lod_r{R}_c{C}) rebuild at varying resolutions — all currently use the same GAS build path
• Cluster terrain partitioning uses BLOCK = min(isqrt(maxVerts)-1, isqrt(maxTris/2)), typically 11 for max 256/256
• _detect_capabilities() in rtx.py queries device cluster support at startup

Design

Distance-Based GAS Type Selection

Zone	Distance	GAS type	Rationale
Near	< threshold₁	Cluster GAS	Best traversal quality for detailed close-up geometry
Far	≥ threshold₁	Standard GAS	Faster build, adequate traversal for simplified meshes

• Threshold aligned with terrain LOD tile grid — entire tiles switch type, not individual triangles
• Default threshold: tile_diagonal * lod_distance_factor (same first LOD transition distance)
• Both GAS types coexist in the same IAS via per-instance transforms

Integration with Terrain LOD

• TerrainLODManager.update() already decides LOD level per tile — extend to also decide GAS type
• LOD 0 tiles → cluster GAS (highest detail, worth the build cost)
• LOD 1+ tiles → standard GAS (reduced triangle count, fast build)
• Tile cache key extends: (tr, tc, lod, base_sub, gas_type)

Integration with Instanced Geometry LOD (#80)

• LOD 0 instances (full-detail mesh) → cluster GAS
• LOD 1+ instances → standard GAS
• Billboard imposters always standard (trivial geometry, 2 triangles)

Rebuild Strategy

• Cluster GAS is expensive — only rebuild when camera moves significantly (> 1 tile width)
• When a tile transitions from standard → cluster, build cluster GAS; cache both versions
• When transitioning cluster → standard, the standard version is already cached or cheap to build
• _rebuild_at_resolution() invalidates both cluster and standard caches (terrain data changed)

Capability Fallback

• If device doesn't support clusters (_detect_capabilities() returns clusters=False), all GAS are standard — no hybrid logic needed
• allowClusteredGeometry pipeline option only set when at least one cluster GAS exists

Implementation Plan

1. GAS type selection in TerrainLODManager — extend update() to tag tiles with gas_type='cluster'|'standard' based on LOD level
2. Dual-path GAS build in rtx.py — add_geometry() accepts optional gas_type parameter, routes to cluster or standard build
3. Cache key extension — add gas_type to tile cache and chunk manager cache keys
4. IAS mixed construction — verify cluster and standard GAS instances coexist correctly in a single IAS (they should — both are valid traversable handles)
5. Threshold configuration — expose cluster_distance_factor on TerrainLODManager and viewer settings
6. Capability gate — skip cluster path entirely on non-cluster-capable devices
7. HUD stats — show cluster/standard tile counts in LOD stats line

Watch Out For

• allowClusteredGeometry = 1 must be set in pipeline compile options whenever any cluster GAS is present — forgetting this causes validation errors / illegal memory access
• 4-param optixGetTriangleVertexData(gas, primIdx, sbtIdx, time, data) works on both cluster and standard GAS — no shader changes needed
• Cluster build memory can be significant for high-detail tiles; monitor GPU memory pressure
• Transition boundary should be a zone, not a hard line, to avoid rebuilding the same tile repeatedly as camera oscillates (reuse hysteresis from Instanced geometry LOD: mesh LOD chains and billboard imposters #80)

Key Files

• rtxpy/rtx.py — cluster accel APIs, add_geometry(), _detect_capabilities()
• rtxpy/viewer/terrain_lod.py — TerrainLODManager.update(), _build_tile_mesh()
• rtxpy/engine.py — _MeshChunkManager, _rebuild_at_resolution()
• rtxpy/lod.py — compute_lod_level(), compute_lod_distances()

References

• Parent issue: Level of detail: terrain LOD, instanced LOD, and hybrid cluster GAS #74
• Instanced geometry LOD: Instanced geometry LOD: mesh LOD chains and billboard imposters #80
• Roadmap: Project roadmap: architecture, rendering, simulation, platform, digital twin #57 (Phase 2 — Level of Detail)
• Terrain LOD PR: Add distance-based terrain LOD system #78