Shader constraints

Shader constraints

Read this before restructuring anything in shaders/trace.hlsl. The walk is shaped the way it is for reasons that are not visible from the code, and two of them are compiler defects rather than design choices.

Both were found the hard way during M9, over a long session in which the symptom was a GPU frame that disagreed with the CPU oracle by a bit-identical amount across a dozen structurally different rewrites. That is the signature of something below the source level.

---

1. One stack push per branch

The rule: any single branch of the walk may push at most one ray onto the stack.

What happens otherwise: fxc silently corrupts the walk's stack arrays. Not just the pushed entries, but the whole array set, including entries written by other branches on earlier iterations. Symptoms observed:

• Pushed rays vanishing entirely (the diffracted orders never rendered).
• Reads of int stack fields coming back as per-pixel noise.
• A branch that was never taken corrupting the results of a different branch below it. Disabling the grating code fixed the polarizer scenes.

The workaround, and now the architecture: the walk is chain and fork. Each ray walks as a chain, every surface continuing it in place, and forks at most one branch. Glass forks its transmitted ray while the reflection continues the chain; a grating forks a revisit of the same surface for its next order while the current order continues the chain. See Architecture § The walk.

The CPU tracer mirrors this exactly. It does not have to, since plain C has no such defect, but a diff between differently-shaped walks would certify nothing.

If you are tempted to push twice: encode the extra state instead. A grating's next-order index rides in the high bits of the depth field (depth | (order << 8)), which costs one shift and no array.

---

2. No dynamically indexed cbuffer arrays for per-object data

The rule: per-object constants read with a runtime index come back as garbage once the constant buffer grows past some threshold.

What happened: adding two more float4 name[8] arrays to the cbuffer, for grating groove directions and order weights, made every read of them return live stack data. Verified by probe: the branch was entered, the rect index was correct, the groove vector was correct, and the weights were noise.

Attempts that did not fix it, all of which are worth not repeating:

• Unrolled compare-select fetch (for j in 0..8: if (j == i) v = arr[j]).
• Hoisting the fetch above the branch condition.
• Compiling as ps_5_0 instead of ps_4_0.
• Splitting the srow stack into four scalar arrays.
• Packing the grating data into unused lanes of existing arrays.

The last one is informative: repacking into rect_glass/rect_filter lanes still failed, which says the problem is the total indexable register pressure, not the array count.

The workaround: grating constants live in scalar (non-array) uniform fields, matched to a hit by comparing the rect index against an index stored in the slot:

float4 grat0_groove_idx;  // xyz groove dir, w rect index or -1
float4 grat0_period_w;    // x period, yzw weights m = -1, 0, +1
float4 grat1_groove_idx;
float4 grat1_period_w;
float4 grat_w2;           // x slot0's +2 weight, y slot1's

Scalar fields are read statically and correctly. The cost is a hard limit of two gratings per scene on the GPU (the CPU tracer has no such limit); a third renders matte black. Raising it means more slots, or a storage-buffer path on a backend that has one.

---

3. pass is a reserved word

HLSL reserves it for effect-framework syntax. A loop written for (int pass = 0; ...) fails to compile, and because the shader is loaded at runtime from a file, the failure surfaces as a crashed executable (0xC0000409), not a build error. The dish intersection uses side instead.

---

4. Shader model 5.0, explicitly

display.c sets d3d11_target to vs_5_0/ps_5_0 rather than accepting sokol's ps_4_0 default. The tracer dynamically indexes both constant buffers and large local arrays; SM4 has no native dynamic constant-buffer indexing and emulates it by copying arrays into indexable temps, which is the same machinery implicated in defect 2.

This did not by itself fix anything, but it removes an entire emulation layer from the picture and is the correct target for a shader of this shape.

---

5. NaN discipline

fxc evaluates both sides of divergent branches. A guard like if (period > 0) { ... lambda / period ... } can still speculate the division with period == 0 and produce a NaN that survives into a select.

Two habits follow:

• Clamp divisors even inside a branch that has already excluded the bad value: float period = max(slot.w, 1e-6);
• Write float tests so a NaN takes the safe path. if (!(rem > 0.0)) return false; treats NaN as evanescent; if (rem <= 0.0) would let it through as a propagating NaN direction.

The oracle also clamps NaN defensively when scoring, so a leak reads as a wrong pixel rather than poisoning the entire mean. If a diff ever reports a mean of -1621797.36, that is what happened.

---

How to debug the next one

The technique that eventually worked, after many that did not:

1. Make the disagreement visible. The oracle dumps both frames on failure. Looking at them side by side ruled out half the hypotheses immediately.
2. Probe with intensity. intensity += 100.0 inside a branch turns "is this code reached?" into a white rectangle. Flag probes (intensity += abs(x - expected) < 0.01 ? 0.0 : 8.0) turn "is this value right?" into a noise field versus a clean one.
3. Suspect the harness before the compiler. A probe that does not change the diff number means your edit is not reaching the GPU. A probe that changes it proves the loop is live and the bug is real.
4. Bisect by disabling, not by rewriting. Replacing the branch condition with if (false) isolated the culprit in one step, after several rewrites had each produced byte-identical failures.

A bit-identical failure across genuinely different source is the tell. Source changes that do not move the number are not being compiled the way you read them.