Fix Interactive Drive Scene Switching and Conditioning Reset

[gtong-nv]

Fix Interactive Drive Scene Switching and Conditioning Reset

Summary

Fixes intermittent interactive-drive scene/variant switch failures where a newly selected variant, such as snow, could race with old queued work or inherit stale world-model conditioning from a previous scene.

This PR makes scene switching generation-safe and conditioning-safe:

• Guards queued scene-load and render commands so stale work cannot bind or render the wrong scene after a newer selection wins.
• Stops the main loop immediately after a scene-change event so it does not enqueue one extra old-scene chunk.
• Resolves weather sibling archives, for example clipgt-...-snow.usdz, to the correct variant/prompt even when the requested variant is still default.
• Clears stale output frames on reset/scene switch so old frames are not presented after a new scene is selected.
• Resets prompt/image conditioning on scene changes while preserving fast manual rollout resets.
• Rebuilds prompt-dependent cross-attention K/V caches after native FP8 DiT releases the full BF16 PyTorch network.

Root Cause

There were two related failure modes.

First, scene selection could race with model warmup and queued work. A stale default scene load or render command could still run after the user selected a newer variant. This made backend logs show a default/no-snow scene and prompt even after the UI printed a snow switch.

Second, native FP8 DiT can release the full BF16 PyTorch network after preparing its optimized FP8 weight snapshot. The weight snapshot is prompt-independent, but cross-attention K/V caches are prompt-dependent. After release, the lightweight shape/cache helper was cloning old cache templates instead of rebuilding cross-attention K/V from the new text context. That allowed visual conditioning, such as snow, to carry across scene switches even when logs showed the new prompt was handed to the backend.

Changes

• Add generation checks before worker-side load_scene and render_chunk calls in ChunkPipeline.
• Clear queued frames when resetting or switching scenes.
• Add backend reset hooks that distinguish manual rollout reset from scene-conditioning reset.
• Make LocalVideoModelAdapter.load_scene() reset scene conditioning before binding the new scene.
• Clear offloaded text/image embeddings when a scene switch requires new conditioning.
• Teach FP8 native DiT shape ops to keep only the small cross-attention projection subset needed after network release.
• Rebuild per-block cross-attention K/V caches from the current prompt context instead of reusing stale template K/V.
• Improve scene-loader variant resolution and scene handoff observability.

Validation

Added regression coverage for:

• Weather sibling archive variant resolution.
• Stale scene loads queued behind model warmup.
• Stale renders queued behind superseded generations.
• Clearing already queued frames on reset and scene switch.
• Scene loads resetting conditioning before first chunk generation.
• Offloaded embeddings rerunning after scene-conditioning reset.
• Native FP8 shape ops rebuilding cross-attention cache from the current context.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gtong-nv]

/ok to test 5bb39eb

[gtong-nv]

/ok to test 585b5c6

[gtong-nv]

/ok to test df440b0