Regression : Disgaea 3 grey color issue

[dbz400]

This affects all backends .Regressed at #2173

[dbz400]

It affected by the depth read texture fetch.

    case rsx::texture_dimension_extended::texture_dimension_2d:
        if (m_prog.redirected_textures & (1 << dst.tex_num))
            SetDst(getFunction(FUNCTION::FUNCTION_TEXTURE_SAMPLE2D_DEPTH_RGBA));
        else
            SetDst(getFunction(FUNCTION::FUNCTION_TEXTURE_SAMPLE2D));
        return true;

[kd-11]

Thats rather odd: the depth read does not alter color.r but only changes color.g and color.b to be tiny non-zero values (so that the combination yields back the original value). I'll need to analyze a trace of this, although I remember texture swizzle problems with this title.
Attach an opengl renderdoc trace and I'll take a look when I can.

[dbz400]

Here it is
rpcs3_the_gate_paradox.zip

[kd-11]

Seems that an RGBA8 RTV is returned as being bound to the depth buffer which should not be possible. Either the game is just being crazy, or there's a bug in our rtt cache code. Redirection should only occur for native depth textures of D24_X8.

[ssshadowww]

This also happens in Atelier Ayesha (and other Atelier games).

renderdock trace: https://mega.nz/#!A5VzRJpB!gfrqI2jl7la_ccKJS4NEOsqpDNENLuk0GkAmqd-cebs

Example:

[kd-11]

Thanks. The bug is our rsx surface cache returning color textures as depth_stencil textures. This can easily be worked around for ogl, but vulkan and dx12 do not attach an internal format to image data, so I'll need to debug this with a sample that is affected to find the real cause.
I'll look through to see if any samples or games that I have are affected by this, since it should be an easy fix.

[kd-11]

@ssshadowww @raven02 Could you run with this build https://ci.appveyor.com/project/kd-11/rpcs3/build/1.0.304 and attach the rpcs3.log for opengl? It'll spam the log with alot of information, but it should help narrow down possible culprits.

[kd-11]

I've updated the link to the appveyor build since it failed to compile the first time

[ssshadowww]

Here is the log for Atelier Ayesha running through the logos:
https://mega.nz/#!5g9lwLxR!vlnwdMLGFnONUCDQRHXxyv4wWy3AtrYb8K5xM3iXSVQ

[dbz400]

Here is the log from debug build on Disgaea 3

http://www.filedropper.com/rpcs3

[kd-11]

The only way these logs are even possible is if we can somehow have a DSV and RTV at the same address! I'm posting a build that should debug this case, and give RTVs priority as that is what the texture cache does.

[kd-11]

https://ci.appveyor.com/project/kd-11/rpcs3/build/1.0.305 should get this one. Please confirm. If it doesnt work, post the new log here.

[kd-11]

Appveyor crashed. New build https://ci.appveyor.com/project/kd-11/rpcs3/build/1.0.306

[dbz400]

Disgaea 3 looks okay again .

@SakataGintokiYT, Is Naruto series also ok?

[kd-11]

If it works it means I was right. Well, now on to #2093

[dbz400]

I run the latest build (https://ci.appveyor.com/project/kd-11/rpcs3/build/1.0.307/artifacts) on Disgaes 3 as well . Log here - RPCS3.zip