vkCmdClearDepthStencilImage causes Metal validation error: PixelFormat xyz is not color renderable

[mellinoe]

I'm attempting to run my graphics library on top of MoltenVK. In it, I clear depth stencil images after they are created so that they can be presented immediately. Essentially, all I am doing is creating a VkImage with format VK_FORMAT_D16_UNORM and depth-stencil, transfer src, and transfer dst usage bits. Then, I create a command buffer and issue a call to vkCmdClearDepthStencilImage where the depth and stencil values are 0.

This generates a Metal validation error. Running under Xcode with "extended" Metal validation gives this information:

validateAttachmentOnDevice:248: failed assertion `PixelFormat MTLPixelFormatDepth16Unorm is not color renderable'

I'm definitely not familiar with the code here at all, but it looks like the procedure for doing this might be trying to use the given texture as a color attachment.

[mellinoe]

I believe I have a fix here: 6517a4d

It was simpler than my previous guess, so there's always the possibility that I'm missing something.

    if (_isDepthStencilClear) {
        _rpsKey.enable(kMVKAttachmentFormatDepthStencilIndex);
+       _rpsKey.attachmentMTLPixelFormats[kMVKAttachmentFormatDepthStencilIndex] =_image->getMTLPixelFormat();

[mellinoe]

Never mind, the above fixes one issue, but there seem to be at least one more in the vicinity. Without my patch above, calling vkCmdClearDepthStencilImage actually gives this validation error:

-[MTLRenderPipelineDescriptorInternal validateWithDevice:]:2400: failed assertion `No valid pixelFormats set.'

Obviously, setting the attachments's pixel format fixes it.

There's some other issue which causes the problem in the issue title, but it seems related as it's in the same function.

[csmartdalton86]

I'm having the same issue:

-[MTLRenderPipelineDescriptorInternal validateWithDevice:]:2400: failed assertion `No valid pixelFormats set.'

With extended validation I get:

validateAttachmentOnDevice:248: failed assertion `PixelFormat MTLPixelFormatStencil8 is not color renderable'

Any luck on finding a solution?

[mellinoe]

I believe the No valid pixelFormats set problem is fixed by #161.

The PixelFormat xyz is not color renderable problem is still happening for me, even on master. Luckily, it's just a debugger validation error, and I don't notice any major rendering artifacts.

[csmartdalton86]

#161 fixes several issues we were having with stencil buffers. Thank you @mellinoe !

I wonder if this issue can be closed now?

[mellinoe]

I believe there's still a bug here somewhere, but unfortunately I haven't had time to debug through everything since a week ago.

When clearing a depth image, MoltenVK seems to use a synthetic render pass (with the appropriate clearing properties set). That synthetic render pass seems to have an issue where it attaches the depth image as a "color target", but with the format set to MTLPixelFormatDepth16Unorm (or whatever other depth format is in use). That triggers the "PixelFormat xyz is not color renderable" error mentioned above. Based on that, I don't think the problem should be too hard to fix, but it was taking me a while to debug through that process, and I gave up after #161 fixed all of my "critical" (aka app-crashing) problems.

There might be more to the issue, but the above is what I learned from my last debugging session.

[billhollings]

@mellinoe

Thanks for all your work on this. Is there anything left to work on here? Can we close this issue?

[mellinoe]

@billhollings The validation error I described above is still happening whenever my app is run under the Xcode debugger. Despite that, I don't see any rendering artifacts or experience any other crashes when running outside the debugger. Does the description I wrote above make any sense, particularly the part about the synthetic render pass used for image clearing operations?

[billhollings]

Fixed in PR #181. Please retest and close.

Further clearing enhancements are under development and will be released soon...but the basic depth clearing should be working now...with no Metal validation errors.

[mellinoe]

Yes, after updating I am completely error-free. Thanks for the fix.