Mesh viewer -> 'VS in' -> preview - Y axis orientation for Vulkan captures

[ZXcolourclash]

Description

Hi, I've been using RenderDoc for Vulkan mesh debugging which has been an absolute Godsend, thank you!

Just one thing I'd like to suggest though, is it possible to invert the Y axis on the Mesh Viewer | VS in preview display?

The reason for this feature request is because I've spent about two days trying to figure out a bug in my Vulkan engine code, and several times I forgot that the Y axis in the previewer is positive up (GL), as opposed to positive down (Vulkan) which means that triangle face direction appears to be reversed. I got quite confused why VK_CULL_MODE_BACK_BIT with VK_FRONT_FACE_CLOCKWISE was culling what I thought were clockwise triangles. It was because the 'clockwise front face' triangles in the VS previewer were actually counter-clockwise front face (the mesh was all CCW front facing triangles).

So I was wondering, if a Vulkan capture was performed, maybe the previewer Y axis direction could be automatically flipped?
Or maybe a dropdown box for Vulkan/GL coord system selection in the VS previewer?

The red, green and blue axis lines could then also show the green line going down indicating a positive down Y axis.

Anyway, thank you so much for a great debugging tool!

Environment

• RenderDoc version: v1.8
• Operating System: Xubuntu 20.04
• Graphics API: Vulkan

[baldurk]

To clarify, no graphics API places any meaning on the input vertex data axis convention because the transformation to vertex output data is entirely up to your shader. It's not accurate to say that GL and Vulkan have different conventions here, since most programs will provide the same mesh data regardless and then perhaps have a different transformation for calculating the output positions.

It's reasonable to account for the possible inversion from the window calculation due to negative viewport height, since RenderDoc's preview is nominally pre-projection after reversing back through NDCs as well, so it would make sense to reverse that as well. That would only apply to the output preview but it would show the correctly applied winding order for the output triangles.

[ZXcolourclash]

I see what you mean about the VS output data winding order depending on the MVP matrix / shader transformations.

I use both the VS in and VS out previews to visually check that the meshes have the correct winding order by rendering a simple mesh (cube) then comparing previews with the mesh vertex data and final window bitmap data. This has proved invaluable in debugging. I don't know if anyone else does this but it works for me! :-D.

I assumed that the VS in previewer is a 3D mesh object space viewer, so that's why I thought it would be useful if the coord system (red, green, blue) indicator should be showing a green line going down for Vulkan captures indicating it's Y positive down in the Vulkan case.

But, I can also see that the VS in preview data doesn't really matter because the MVP shader matrix transformation and final VS out data is what matters in terms of viewport transform, rasterisation etc.

[baldurk]

Yes exactly, the VS input preview is an object space viewer. That's what I was trying to clarify - what you're saying about vulkan having any convention of "down" being Y positive in object space is definitely not correct. Vulkan only differs in its viewport transform from NDC co-ordinates, where by default it doesn't do any flip meaning that the Y-positive axis in NDC maps to "down" in the viewport.

That has no bearing on whatever object space convention you choose though, which is most typically either Y up or Z up. From there the application either bakes in a Y flip into its projection matrix, manually flips the Y of the Position builtin at the end of the vertex shader, or uses the negative viewport height extension to avoid the need for that entirely.

That commit will account for the viewport flip in the VS output preview, which is the only change that makes sense to make.