Make sure blend is disabled if render target has integer format

[gdkchan]

Having blend enabled for integer formats is invalid. The most common case where this happens is when logic operations are enabled, and in this case the blend is ignored. Since Metal/MoltenVK does not support logical operations, it is never enabled there, and having blend enabled in this case causes a Metal assert and crash. To avoid this, this change forces blend to be disabled if the respective framebuffer attachment is using a integer format.

This should be safe to enable for all vendors, but here I'm enabling it only on MoltenVK just to be on the safe side. But I can enable it for all vendors too.

Fixes crashes on Xenoblade games, and should fix also fix some crashes on Luigi's Mansion 3 (on macOS). The Luigi's Mansion 3 crash would usually happen when entering the bathroom on Luigi's room for example.

Contributes to #4062.

[github-actions]

Download the artifacts for this pull request:

• ryujinx-Release-1.1.0+9a514b8-linux_x64
• ryujinx-Release-1.1.0+9a514b8-win_x64

Experimental GUI (Avalonia)

• ava-ryujinx-Release-1.1.0+9a514b8-linux_x64
• ava-ryujinx-Release-1.1.0+9a514b8-macos_universal
• ava-ryujinx-Release-1.1.0+9a514b8-win_x64

GUI-less (SDL2)

• sdl2-ryujinx-headless-Release-1.1.0+9a514b8-linux_x64
• sdl2-ryujinx-headless-Release-1.1.0+9a514b8-win_x64

Only for Developers

• ava-ryujinx-Debug-1.1.0+9a514b8-linux_x64
• ava-ryujinx-Debug-1.1.0+9a514b8-macos_universal
• ava-ryujinx-Debug-1.1.0+9a514b8-win_x64
• ryujinx-Debug-1.1.0+9a514b8-linux_x64
• ryujinx-Debug-1.1.0+9a514b8-win_x64
• sdl2-ryujinx-headless-Debug-1.1.0+9a514b8-linux_x64
• sdl2-ryujinx-headless-Debug-1.1.0+9a514b8-win_x64

[marysaka]

lgtm

[riperiperi]

That's sensible. Does this cause a validation error normally or is it just ignored?

[gdkchan]

That's sensible. Does this cause a validation error normally or is it just ignored?

Yes, it has this validation error:

Validation Error: [ VUID-vkCmdDrawIndexed-blendEnable-04727 ] Object 0: handle = 0x4c8e060000040ccd, type = VK_OBJECT_TYPE_PIPELINE; | MessageID = 0x8a06751e | vkCmdDrawIndexed: Image view's format features of the color attachment (0) of the active subpass do not contain VK_FORMAT_FEATURE_COLOR_ATTACHMENT_BLEND_BIT bit, but active pipeline's pAttachments[0].blendEnable is not VK_FALSE. The Vulkan spec states: If rasterization is not disabled in the bound graphics pipeline, then for each color attachment in the subpass, if the corresponding image view's format features do not contain VK_FORMAT_FEATURE_COLOR_ATTACHMENT_BLEND_BIT, then the blendEnable member of the corresponding element of the pAttachments member of pColorBlendState must be VK_FALSE (https://vulkan.lunarg.com/doc/view/1.3.231.1/windows/1.3-extensions/vkspec.html#VUID-vkCmdDrawIndexed-blendEnable-04727)