RTAO: fix denoiser's unsafe wave-lane ↔ SV_GroupIndex ordering assumption - #967
Merged
amarpMSFT merged 1 commit intoJul 16, 2026
Merged
Conversation
The CalculateMeanVariance and DisocclusionBlur compute shaders exchanged each row's values across wave lanes while deriving target lane indices from SV_GroupIndex. For their 2D [numthreads(8,8,1)] groups, HLSL does not guarantee any relationship between SV_GroupIndex and WaveGetLaneIndex. The shaders also assumed a wave width of at least 16 without enforcing it. On WARP (wave width 4), the wave reads address nonexistent or unrelated lanes and corrupt output. Replace those exchanges with groupshared-memory caches indexed by the logical thread layout. This is independent of wave size and thread-to-lane assignment. Apply the same correction to the unused depth-aware separable gaussian variant and rename it now that it no longer uses WaveReadLaneAt. Remove stale measured and relative performance claims from the README and the old shader timing comment because they describe the previous implementation. Validation: - dxc cs_6_3 compiles all three groupshared shaders. - Their DXIL is byte-identical to the groupshared binaries used for the WARP A/B tests. - CalculateMeanVariance groupshared output matches a CPU reference to half-precision (max mean delta 0.000338, max variance delta 0.000296), while the old wave path was wrong on 576/576 tested interior pixels on WARP. - The depth-aware gaussian groupshared output matches its CPU reference, while the old wave path was wrong on 35/36 tested interior pixels on WARP.
amarpMSFT
force-pushed
the
user/amarp/rtao-wave-lane-ordering
branch
from
July 16, 2026 03:36
5357677 to
c09d844
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The RTAO denoiser's
CalculateMeanVarianceCSandDisocclusionBlur3x3CScompute shaders exchanged each row's values across wave lanes while deriving source-lane indices fromSV_GroupIndex. That assumes row-major thread-to-lane packing and a wave width of at least 16, neither of which HLSL guarantees for these 2D thread groups.This PR removes that assumption. The affected shaders now exchange values only through groupshared-memory caches indexed by their logical thread coordinates. There is no optional wave path or runtime capability gate.
Why it's a bug
Both live shaders use 2D thread groups (
[numthreads(8, 8, 1)]) and previously did the equivalent of:This returns the intended neighbour only if the implementation maps each thread's wave lane index to its row-major
SV_GroupIndex. The HLSL SM 6.6 derivatives specification explicitly states that for 2D thread groups "no relation between the values ofSV_GroupIndexand the return value ofWaveGetLaneIndex()should be assumed." Wave-op support and wave width cannot establish that ordering.Repro
On WARP (wave width 4), the old shader path reads nonexistent or unrelated lanes. A D3D12 compute harness ran the affected shaders against CPU references:
CalculateMeanVariance: the old wave path was wrong on 576/576 tested interior pixels. The groupshared implementation matched to half precision (maximum mean delta0.000338; maximum variance delta0.000296).The fix
CalculateMeanVarianceCS.hlslnow loads the logical 16x16 input tile into groupshared memory, synchronizes the group, and calculates each horizontal kernel from that cache.DisocclusionBlur3x3CS.hlslnow does the equivalent groupshared exchange for each(value, depth)pair before applying the depth-aware gaussian weights.DepthAwareSeparableGaussianFilter3x3CS.hlslbecause it no longer usesWaveReadLaneAt.Validation
dxcascs_6_3.WaveReadLaneAtcall or deleted fast-path identifier remains in this sample.origin/master.git diff --checkpasses, and the project XML files parse successfully.Notes for reviewers
The earlier capability-gated wave permutation has been removed rather than retained behind
WaveOps/WaveLaneCountMin: those feature values describe intrinsic availability and width, not lane-to-thread ordering. Groupshared exchange is now the sole implementation, so correctness is independent of wave size and lane assignment.