Allow skipping draws with broken pipeline variants on Vulkan

[gdkchan]

Currently we skip draws when the background compilation of the shaders on Vulkan failed. This is detected with the result code from the pipeline creation call. If an is error returned, we assume that the host failed to compile the shader that the emulator gave. While this generally works, there is an odd situation on macOS with MoltenVK where some pipeline variants may compile successfully, and others may fail. This means that the emulator can still crash in some cases due to shader compilation failures.

This PR allows draws to be skipped when any pipeline variant creation fails, not just the background one. This is only done for graphics shaders since compute generally doesn't have pipeline variants (since the only pipeline state on compute is the shader itself). This also includes minor improvements to try preventing any performance regression from additional checks. The !_program.IsLinked has been moved to CreatePipeline to avoid checking it all the time. RecreatePipelineIfNeeded has been split in two, one for graphics and other for compute. Most of what it was doing is not really necessary for compute (like updating dynamic state, vertex/index/transform feedback buffers).

Fixes #5790, although it does not fix the underlying issue (the compilation error). The only error that the compiler produces here is "Compiler encountered an internal error.", which does not really give much indication of what is wrong. I was able to track down the specific pipeline state that makes it produce that error, it was ColorBlendAttachmentState, on the background pipeline it was disabled, on the new one it is enabled. There is a single color attachment with VkFormat B10G11R11UfloatPack32.

Does not render 100% due to the skipped draw, but it is better than the game crashing.

Testing is welcome, also to ensure that nothing broke on platforms other than macOS.

[github-actions]

Download the artifacts for this pull request:

• ryujinx-Release-1.1.0+b50522b-linux_x64
• ryujinx-Release-1.1.0+b50522b-win_x64

Experimental GUI (Avalonia)

• ava-ryujinx-Release-1.1.0+b50522b-linux_x64
• ava-ryujinx-Release-1.1.0+b50522b-macos_universal
• ava-ryujinx-Release-1.1.0+b50522b-win_x64

GUI-less (SDL2)

• sdl2-ryujinx-headless-Release-1.1.0+b50522b-linux_x64
• sdl2-ryujinx-headless-Release-1.1.0+b50522b-macos_universal
• sdl2-ryujinx-headless-Release-1.1.0+b50522b-win_x64

Only for Developers

• ava-ryujinx-Debug-1.1.0+b50522b-macos_universal
• sdl2-ryujinx-headless-Debug-1.1.0+b50522b-macos_universal
• ryujinx-Debug-1.1.0+b50522b-linux_x64
• sdl2-ryujinx-headless-Debug-1.1.0+b50522b-linux_x64
• ava-ryujinx-Debug-1.1.0+b50522b-linux_x64
• ryujinx-Debug-1.1.0+b50522b-win_x64
• sdl2-ryujinx-headless-Debug-1.1.0+b50522b-win_x64
• ava-ryujinx-Debug-1.1.0+b50522b-win_x64

[marysaka]

Really not a big fan of this, but it will do for now I guess...

[gdkchan]

Really not a big fan of this, but it will do for now I guess...

I guess the title makes it sound like I'm introducing a hack on Vulkan to work around MoltenVK/Metal issues, but this just extends something that is already being done. We already skip draws when shader compilation fails on OpenGL, and do more or less the same on Vulkan, but only when the first, background pipeline creation fails. This is just extending it to work on all pipelines. I would prefer to fix the underlying issue, but I think there is some bug on their compiler since it just reports an "internal compiler error", it might be possible to work around it, but it would require either a lot of trial and error or reversing their compiler (maybe not that hard since it's LLVM based, but still).

[psnluiz]

This fixed Fire Emblem Three Houses crashing for me.

[riperiperi]

Looks good to me. What was the expected extra cost of adding this? I can't seem to find anything that would cost too much more.

[gdkchan]

Looks good to me. What was the expected extra cost of adding this? I can't seem to find anything that would cost too much more.

There are a few extra checks needed for updating pipelines, but IIRC I tried to make up for it (for example by moving the shader IsLinked check).