Disable Wayland for RenderDoc

[abocado18]

I want to use Renderdoc to debug my application, but I get Diligent Engine: ERROR in Instance() (Instance.cpp, 420): Required extension VK_KHR_wayland_surface is not available when starting it. Can I somehow disable it?

[fahadiot]

Not at runtime. On Linux DiligentCore is always built with VK_USE_PLATFORM_WAYLAND_KHR (together with XCB and Xlib), and Instance.cpp then pushes VK_KHR_wayland_surface into the required instance extensions and throws if it isn't there. There's no CreateInfo flag or CMake option to turn that off.

The reason RenderDoc trips on it is that official RenderDoc builds don't expose VK_KHR_wayland_surface at all. Wayland support there is behind the ENABLE_UNSUPPORTED_EXPERIMENTAL_POSSIBLY_BROKEN_WAYLAND build option, which is off by default.

The simplest workaround is to drop the Wayland define from your local DiligentCore and rebuild:
https://github.com/DiligentGraphics/DiligentCore/blob/5bbbc0b6e14458604aa7e90e91919d27cdd23dcb/BuildTools/CMake/VulkanUtils.cmake#L8-L12

Just remove the VK_USE_PLATFORM_WAYLAND_KHR=1 line. The Wayland surface code in SwapChainVkImpl.cpp and the extension in Instance.cpp are both behind that define, so they compile out cleanly. Your app then has to create its window through X11 (XWayland is fine), which is what RenderDoc needs anyway to capture it.

[abocado18]

Thank you, i can capturre it now

[TheMostDiligent]

DiligentGraphics/DiligentCore@560ffe3 should fix this - all extensions were enabled unconditionally, now they are only enabled when supported.