[Bug] RectLight isProjector pattern only resolves when the camera is stationary (RTPT radiance cache)

[junminkyung]

Description

Topic Description

RectLight used as a pattern projector (isProjector + projector:directionTexture:file) only
resolves its dot pattern while the camera is nearly stationary under RealTimePathTracing.
Once the camera moves, the dots wash out into a flat, uniform brightening. This appears to be
radiance-cache convergence rather than motion blur.

Detailed Description

I am simulating an active-stereo depth camera. A small UsdLux.RectLight acts as the dot
projector, rigidly attached next to a stereo IR camera pair on a mobile robot. The projected dots
are what the stereo matcher keys on, so the pattern has to be present in every frame, including
while the robot drives.

Projector setup:

light = UsdLux.RectLight.Define(stage, light_path)
prim = light.GetPrim()
UsdLux.ShapingAPI.Apply(prim)
light.CreateIntensityAttr(10.0)
light.CreateWidthAttr(0.005); light.CreateHeightAttr(0.005)
prim.CreateAttribute("isProjector", Sdf.ValueTypeNames.Bool).Set(True)
prim.CreateAttribute("visibleInPrimaryRay", Sdf.ValueTypeNames.Bool).Set(False)
prim.CreateAttribute("inputs:texture:file", Sdf.ValueTypeNames.Asset).Set(dot_pattern_png)
prim.CreateAttribute("projector:directionTexture:file",
                     Sdf.ValueTypeNames.Asset).Set(octahedral_direction_lut_exr)

Cameras are omni.replicator.core render products at 1280x800 with the LdrColor annotator,
ticked at 30 Hz. Scene rendermode is RealTimePathTracing (set by the environment USD).

Expected: the dot pattern is rendered every frame regardless of camera motion.

Actual: the pattern is only present when the camera is nearly still. While driving it is
replaced by a smooth, brighter wash. Energy looks conserved — on the floor the median brightness
rises from 63 to about 100 DN while the local contrast (median 9x9 standard deviation) collapses
from about 13 to 1.6.

Across one full 1200 s recording (932 IR frames), only the five frames captured before the drive
trigger carry dots:

segment	frames	floor dot contrast (median 9x9 std)
before the drive trigger (stationary)	5	21 to 49
driving at 0.4 m/s	927	1.6 (0.5 % of frames above 20)

Speed sweep, constant-velocity straight line, identical scene and settings:

speed	floor dot contrast (median)	frames with contrast > 20
0.0 m/s	27.9	63.9 %
0.05 m/s	6.4	13.9 %
0.4 m/s	1.6	0 %

While stationary the contrast keeps climbing over roughly one second of simulated time
(27.9 at 0.6 s, 39.2 at 1.2 s). That rising trend, rather than a fixed blur, is what points at an
accumulation/convergence effect. Near surfaces (a car underside 0.2 to 0.3 m away) keep their dots
even while driving; it is the far, grazing-incidence floor that loses them.

With the scene loaded, these read as follows — i.e. the obvious accumulation switches are already
off, which is why the behaviour is surprising to me:

/rtx/pathtracing/temporalAccumulation = false
/rtx/raytracing/enableAccumulation    = false
/rtx/post/motionblur/enabled          = false
/rtx/pathtracing/spp = totalSpp = clampSpp = 64
/rtx/rtpt/cached/enabled              = true
/rtx/rtpt/cached/updateSampleCount    = 1
/rtx/rtpt/cached/rcTileSize           = 5

The environment USD's /Render prim does not override motion blur or rendermode.

Steps to Reproduce

1. Load any scene whose rendermode is RealTimePathTracing, containing a floor seen at grazing
   incidence by a camera mounted low (about 0.12 m above the floor).
2. Create a UsdLux.RectLight as above (isProjector, visibleInPrimaryRay=false, a dot-pattern
   texture and an octahedral direction LUT), parented next to the camera so it moves with it.
3. Create a render product for that camera with the LdrColor annotator at 30 Hz and start the
   timeline with the camera stationary. Wait about 1 s of simulated time — the dot pattern appears
   and its local contrast keeps rising.
4. Drive the camera forward at 0.4 m/s. Within about 0.5 s of simulated time the dots disappear and
   the surface brightens uniformly.
5. Measure the median 9x9 local standard deviation over the lower rows of the image to see the
   collapse numerically.

Error Messages

None. Nothing is logged; the render simply loses the pattern. No texture-load failure and no
primvar warning appears for the projector or its textures.

Screenshots or Videos

Not attached here. The effect is reproduced numerically by the contrast tables above; if it would
help I can attach stationary and driving frames from the same scene.

---

Additional Information

What I've Tried

Each setting below was applied before the render products were created and confirmed applied in
the log. Floor dot contrast while driving at 0.4 m/s (stationary reference is 27.9):

change	contrast
baseline (RealTimePathTracing)	1.6 to 2.0
/rtx/rendermode = RaytracedLighting	1.93
/rtx/post/aa/op 3 (DLSS) to 2 (FXAA)	2.04
/rtx/directLighting/sampledLighting/enabled = false	1.91
rendering rate 2800 Hz to 60 Hz	1.60
/rtx/pathtracing/optixDenoiser/enabled = false	1.67
/rtx/lightspeed/NRD_ReLAX/{diffuse,specular}HistoryFrames 31 to 1	1.89
/rtx/pathtracing/spp and totalSpp 64 to 1	1.90
/rtx/rtpt/cached/updateSampleCount 1 to 32	2.02
/rtx/rtpt/cached/rcTileSize 5 to 1	3.74
/rtx/rtpt/maxRoughness 0.3 to 1.0 (+ modulatingRoughnessThreshold 1.0)	2.27
/rtx/rtpt/cached/terminatePathOnFirstDiffuseHit = false	1.92
/rtx/rtpt/fireflyFilter/enabled = false	1.90
/rtx/rtpt/cfgpuCameraJitter = false	1.95
/rtx/rendermode = PathTracing	52.4 (100 % of frames above 20)

Two further observations that made me suspect the cached path:

• With /rtx/rtpt/cached/enabled = false the dots never appear at all, not even while
  stationary (first-frame contrast drops from 12-15 to 2.2). So the pattern seems to be produced
  through the cached path rather than in spite of it.
• PathTracing holds the pattern perfectly while driving, but at 1280x800 it produced only
  10 frames in 180 s of wall clock in this scene, so it cannot be used for continuous capture.

Questions

1. Is this expected behaviour of the RealTimePathTracing radiance cache — that a high-frequency
   projected pattern needs roughly a second of a static camera to resolve, and that camera motion
   invalidates it faster than it converges?
2. Is there a supported way to make a projector's contribution bypass the cache, or to force the
   cache to converge within a single frame, while staying in a real-time render mode? The knobs I
   found (updateSampleCount, rcTileSize, terminatePathOnFirstDiffuseHit, maxRoughness) each
   moved the number slightly but never restored the pattern.
3. Is there a recommended approach for simulating an active-stereo / structured-light projector on
   a moving platform in Isaac Sim? StructuredLightCamera in
   isaacsim.sensors.experimental.rtx looks related — is it subject to the same limitation?

Related Issues

I did not find an existing forum topic or GitHub issue describing projector patterns being lost
under camera motion. If one exists I would appreciate a pointer.

Additional Context — the workaround I settled on

In case it helps someone hitting the same wall: I stopped projecting light and started computing the
pattern instead. Each stereo IR camera already has a distance_to_image_plane annotator, so per
pixel I reconstruct the world point, transform it into the projector frame, sample the dot texture
at the resulting angle, and add gain * pattern * cos(incidence) / distance^2 to the IR image.
Both cameras use the same projector pose, so the same world point receives the same dot and the
resulting disparity is physically consistent — which is what the stereo matcher needs.

It is independent of renderer timing, costs about 10 % of the IR frame rate, and as a side effect
the dots no longer contaminate the RGB camera (a real IR projector is invisible to an IR-cut RGB
sensor, but an Isaac light has no wavelength).

That solves my use case, but it is a workaround. I would still like to understand the cache
behaviour, because anything that relies on fine projected detail on a moving platform will hit it.

Isaac Sim version

6.0.1

Operating System (OS)

Ubuntu 22.04

GPU Name

RTX 3090 Ti x 2

GPU Driver and CUDA versions

580.159.04

Logs

No response

Additional information

No response

[junminkyung]

Resolved: the cause was DLSS, not the radiance cache. Leaving the data here in case it helps someone else.

Switching anti-aliasing from the default DLSS to DLAA restores the projected pattern under camera motion. Everything else unchanged (RealTimePathTracing, same RectLight isProjector rig, same scene, same 0.4 m/s drive):

/rtx/post/aa/op	floor dot contrast while driving (median 9x9 std)	frames holding the pattern
3 = DLSS (default)	1.6 - 2.0	0 %
2 = FXAA	2.04	0 %
0 = none	2.04	0 %
4 = DLAA	26.51	93.1 %

With DLAA it is speed-independent: 33.22 stationary, 26.51 at 0.4 m/s, 27.62 at 0.8 m/s. Stationary reference with DLSS was 27.9, so DLAA at speed matches the stationary case.

My reading is that DLSS renders below native resolution and upscales, so a fine dot pattern is undersampled and its temporal component smears it under motion, while DLAA runs the same network at native resolution. I have no explanation for why FXAA and no-AA do not recover it — that is the part I still do not understand, and it is what sent me down the radiance-cache path.

Two follow-up questions, if a maintainer has a moment:

1. Is DLAA the recommended anti-aliasing mode when a render product has to preserve high-frequency projected detail (structured light, IR dot projectors)? It would be useful to have this stated in the StructuredLightCamera docs, which currently say nothing about render settings.
2. Is StructuredLightCamera expected to work under continuous camera motion, or only with stepped rep.orchestrator.step(rt_subframes=N) capture? I tried rt_subframes (8 and 32) in a free-running loop that reads annotators directly and no frames were produced at all in a 180 s window, so it appears to require the orchestrator stepping path.

For context on why this matters: I am simulating an active-stereo depth camera where the projected dots are the signal the stereo matcher keys on, so the pattern has to survive continuous driving at 0.4 m/s. With DLAA the resulting depth now matches the real sensor closely — invalid rate 52.9 % against a real 53.6 %, and the vehicle surface 3.8 % invalid against a real 8.4 %.

[zwdoescode]

Thanks for your detailed report and I am glad you found a resolution! About your follow-up questions, let me reach out to the internal team as it is not well documented.