Pixel History

Sketched implementation details for D3D11

This is a 'final' implementation with all the functionality we'd want. There would be value in doing partial implementation that e.g. at a minimum just showed you which drawcalls modified the target (without listing values or if there are draws that would have written had they not failed some tests).

1. Determine at a high level the subset of draws that could possibly write to the selected target and pixel, this should cut down the workload.

2. Run through the log for each drawcall (ideally doing this incrementally to save a lot of time, otherwise O(n^2) kills perf - see the post-shader mesh fetching when it's done over a batch of draws).

3. Render with a viewport or scissor set to only the target pixel. Remove any other checks like depth testing, backface culling, etc.

   Replace the pixel shader with a fixed-colour shader and do occlusion queries on each candidate draw to determine which draws rasterize to that pixel.

4. Hopefully this should produce a smaller set. For this set of draws we run a series of iterative tests as below.

5. Run the draw with all relevant tests disabled with the real pixel shader and retrieve the output colour and depth - if there's no output then the pixel shader issued a kill, note this instead.

6. Run the draw with all relevant tests enabled to see if it fails any of them. If it does fail, binary chop or just iteratively re-enable tests to see which one it failed, and note that (e.g. "pixel culled due to backface culling" or whatever).

The difficulties that remain with this are:

• Retrieving the output colour and depth. If a draw overdraws the pixel more than once (likely) then we want to get each individual fragment that's output. We could do this with UAV writes but we want to maintain the existing pixel shader. In theory you could translate the bytecode back to HLSL and append the appropriate instructions to write to a UAV but that's very complex.

  We can get the final result by just rendering to a new render target but this doesn't account for overdraw or blending. One potential solution to this would be to use stencil testing to count the number of layers we go through and set the stencil value to the 'layer' we want to retrieve (similar to stencil shadowing techniques). This does mean N iterations for the drawcall though so might be slow. It should work though as the order of fragments written to the screen should be contractually guaranteed.

• Handling MSAA - haven't really thought about this yet.

• It would also be awesome to get the primitive number out for each fragment, we can then work backwards to get the vertices that made up the triangle that was rasterized. If no geometry shader or tessellation is bound then the pixel shader can just bind a semantic to obtain the primitive ID, but in other cases this isn't possible.

  The way I've thought to do this would be to fetch the post-shader mesh for this drawcall and upload the positions to a custom vertex shader that just does a pass-through and appends the primitive ID with this. However that only gives you the primitive output from the final stage - that doesn't translate through to a primitive (or control point set) from the initial data read to the vertex shader. Perhaps that's not important as it's so user-defined?

The key will be whether or not this amount of work would be prohibitive for a reasonably response time to the user (I think anything more than 1-2 seconds would be unacceptable). With optimisations to log replaying perhaps it becomes more tractable, but the hope is that reducing the drawcall set as much as possible will provide most of the benefit as we then only need to perform whichever iterations are necessary for the matching draws (depends entirely on the overdraw on that target).