Crash when inspecting draw command when using custom compiler

[heroseh]

Description

Hey, I have encounter a crash when loading any of the shaders that my compiler hcc creates.

Long story short, I have found that the issue is coming from how my compiler emits OpVariable with an Output specifier that is a structure type. All other compilers that i have found, create Output OpVariable with scalar types only. I have verified that this is the case by stripping out the structure type Output OpVariable's completely and all the code that references these from the spirv file using a hex editor.

          %3 = OpVariable %_ptr_Output__struct_150 Output
          %5 = OpVariable %_ptr_Output__struct_152 Output
        %103 = OpVariable %_ptr_Output__struct_102 Output
        %144 = OpVariable %_ptr_Output__struct_143 Output

I have uploaded a zip file containing the following files:

• shader-causing-crash.spirv
  • this is a vertex and fragment shader that renders a full screen quad using the interpolated UV X & Y as fragment R & G
• shader-modified-to-work.spirv
  • this is the same shader but strips out all all Output OpVariable with structure types and the code the references them
  • since this does not write out a vertex or fragment, it will have a black screen and that is expected
• hcc-renderdoc-causing-crash.rdc
  • renderdoc capture that crashes when you click to inspect the single vkCmdDraw
• hcc-renderdoc-modified-to-work.rdc
  • renderdoc capture that uses the modified spirv file that does not crash when inspecting the single vkCmdDraw
  • since this does not write out a vertex or fragment, it will have a black screen and that is expected

hcc-renderdoc-crash.zip

This is was captured using Linux but i won't be surprised if this happens on Windows.

Funnily enough the AMD Windows driver also crashed because of this issue too, so if you are debugging this on Windows, avoid using AMD hardware as it will crash at vkCreateShaderModule and not at the code in renderdoc that is causing the issue.
forum post, github

Thank you for looking into this, please let me know if you need anything else from me.

Steps to reproduce

1. download & unzip hcc-renderdoc-crash.zip
2. open hcc-renderdoc-causing-crash.rdc (found in the extracted zip folder) capture in renderdoc
3. click to inspect the single vkCmdDraw in the Event Browser window and observe the crash

Environment

• RenderDoc version: 1.26 (built from unknown source) (QT version 5.15.8)
• Operating System: Manjaro
• Graphics API: Vulkan 1.3
• Graphics Card: AMD RX 6600

[baldurk]

I believe that commit should fix it. Output structs are pretty common and glslang will generate them easily, but an input struct to the vertex shader with builtins was causing problems, though I don't believe the spec disallows it as it allows it on outputs.

[heroseh]

Thank you for fixing that so quickly, really appreciate it. I downloaded the latest code and can verify it fixes the zip file i provided above.

I wasn't sure if i should create a new ticket for this but i thought it might be very closely related to this issue.

I have tried the fix on more of my shader samples and it is crashing in the same place, and when i fully commented out the vertex shader function body of the triangle.c sample, it worked. But i also have a compute shader that is crashing in the same place too. So i thought i would collect together 4 of my samples each with a source file, spir-v file and renderdoc capture.

you may find the 4 sample files here: hcc-renderdoc-crash-2.zip

Thank you!

[baldurk]

Looks like a similar issue when fetching bindless feedback, fixed in that commit. I didn't get a crash on the triangle.rdc in the zip you sent, only in the others that used resources.