Rendering issues in Tantalus Media games (Spongebob, MX ATV, etc)

[hrydgard]

As Gamemulatorer pointed out in #9018 (comment) , these games have weird rendering techniques in common.

All except MegaMind (and it seems MX vs ATV: On The Edge) render a very complicated striped post-processing effect, that doesn't actually seems to do very much (EDIT: it applies textured depth fog).

The details seem to vary a bit from game to game but all have in common copying the screen in small stripes to a temporary and doing strange processing there, involving depth texturing and reinterpret between 16 and 32-bit formats. I created this common issue where I'll write up any progress and link PRs to.

We are currently very unsuccessful emulating this, but I think it's not quite as hard as it looks.

Features I think we need to implement for this to work:

• Reimplement framebuffer reinterpret on the new color copy framework #15892
• Detect use of a framebuffer living in the "stride gap" of another one and adjust its width accordingly (and make sure it doesn't get detected as the same texture as the big one)
• Probably smarter interactions between block copies and framebuffers, taking the new sequence numbers into account etc. In the end, we might to share framebuffer match detection logic between block copies, textures and framebuffers...
• 32-bit<->16-bit format reinterpret: Reinterpret between 32 and 16 bit texture formats #15907

Still will likely not perform very well, but better than nothing.

Details are in the various sub-issues:

• White overlap in Cars Race-o-Rama #9850
• MX vs. ATV Reflex (USA) ULUS10429 graphics problem #9604
• Rendering issue in Spongebob Yellow Avenger  #9018

[Panderner]

It's seems MX vs. ATV On the Edge working fine

[hrydgard]

I'm starting to understand the idea of the effect in Spongebob, although I'm currently getting this:

The whole idea is to get depth into the alpha bits of the main RGBA8888 framebuffer, so that at the end, a background image can be blended to it using destination alpha, giving a "textured fog" effect (in the distance, things will fade into the image instead of just a single color).

The screen is processed in 16-pixel-wide vertical strips, first every other strip, then it fills in with the second set.

There's a special strip buffer, 16 pixels wide, which is used both as an 8888 buffer or a 32-pixel-wide 565 buffer as required.

First, depth is copied into the strip and stretched twice as wide, using CLUT16 depal texturing from Z. There seems to be something going wrong here, since we get a quickly repeating gradient instead of a single one. Anyway, the depal puts the gradient into blue and red bits of each 16-bit, as follows:

16-bit 2 pixels:
BBBBBGGGgggrrrrrBBBBBGGGgggrrrrr  where capitals are set bits
32-bit 1 pixel:
AAAAAAAArrrrrrrrGGGGGGGGbbbbbbbb

When reinterpreted as halfwidth 32-bit 8888, these blue and green bits end up in alpha (!) and green (ignored), as seen above.

Next, the main color buffer is drawn on top, writing only to R G and B. As a result, in our thin stripe, we now have A=depth while RGB comes from the main buffer.

This stripe is now copied back into position in the main image using a block transfer.

Repeat until the whole image is processed. One more pass is done across the screen whose purpose I haven't yet determined (but it seems to only affect sky pixels), then as stated above, a large image is blended on top of the screen, with blend mode set t DST_ALPHA / ONE_MINUS_DST_ALPHA.

As can be seen in the above screenshot, not all of that is working perfectly, but the process is now partially working, just need to figure out the remaining issues...

[hrydgard]

Just noticed that the depth texturing is done from 0x288000, not 0x688000 which we might have expected if the game used the full swizzle from +6MB. So the game is actually reading the depth buffer in the +2MB format seen here, where 32-pixel-wide columns are swapped around: http://hitmen.c02.at/files/yapspd/psp_doc/chap10.html#sec10.2

Maybe they just didn't know about the full swizzle at +6MB?

Either way, that means that we have to emulate the column-swapping swizzle during depth depal.

[hrydgard]

I think we can conclude that on the PSP, to find the corresponding Z pixel from a FB pixel coordinate when rasterizing, the following transforms are applied, forming the "Swizzle":

1. Bits 4-8 of the X coordinate are rotated one step (only when color is 32-bit)
2. The Y coordinate is xor-ed by 3

These are probably chosen to be cheap to implement and minimize cache hierarchy collisions between color and depth writes/reads.

Since this was annoying when doing depth effects, hardware was added to the read path, to:

• if address & 0x600000 == 0x200000, do 2 in reverse
• if address & 0x600000 == 0x600000, do 1 and 2 in reverse

Since 1 only happens in 32-bit color mode, this is why we see 0x200000 addresses in games that render in 16-bit color, and 0x600000 addresses in games that render in 32-bit. But not here, it's using 0x200000 - apparently the modes were not well documented, since games try to apply parts of the swizzle on their own.

Since we don't store our Z buffers swizzled, we simply have to reverse these when the two address rules would have been applied.

Link to the "doc": http://hitmen.c02.at/files/yapspd/psp_doc/chap10.html#sec10.2

[hrydgard]

Most of the swizzle seems to be right. Still having the weird issue with the lookup.

[hrydgard]

Note to self: The missing stripe is because due to the swizzle, we need to expand the depal surface all the way out to 512 (or beyond?) in width, since the last column won't be in the expected place...

[hrydgard]

That's better.

Now only the weird color ramp repeat left... I'm stumped there.

[hrydgard]

On real hardware, the gradient is not actually visible, at least not in this scene, maybe because it's very small at the horizon. If I change the code to clamp instead of wrap during the lookup, I get an image that matches hardware.

However, that's just a wild guess. Think I'm gonna have to dust off the PSPSDK and write a hardware test...

[hrydgard]

No, I think there's another step that fails. Towards the end, just before blending the background, the game attempts to initialize the stencil buffer from a depth test, I think (red/green mask is depth test results in RenderDoc):

So since alpha/stencil are shared on the PSP, it tries to this way zero out stencil on pixels that are closer to the camera, to eliminate this repeating effect when out of range for the distance gradient.

In PPSSPP, the stencil buffer is already zero since the reinterpret/copy shenanigans didn't set it to anything. However that doesn't really matter, it's not tested against. What does matter is that the Depth Fail Op -> Zero needs to land in our destination alpha (it's read in the subsequent pass), and it doesn't, obviously. This is nasty to emulate since depth fail = the pixel is dead and doesn't do anything.

This is a bit similar to #15813 but worse.

We might need to map this to a custom shader doing a manual depth test, writing zero to alpha if failing...

EDIT: Actually, we can fake this by flipping the depth test around, I think.

[hrydgard]

Yup, that did it.

Still some brokenness to debug on OpenGL and D3D9, but Vulkan and D3D11 look perfect now in Spongebob.

The other games are still broken, but they use variations on the same technique, so with some additional work, should also be fixable.

[hrydgard]

Cars' solution for the same thing does work on PC GPUs already, it already flips it around to the easy way. However it breaks earlier in a different way, we are texturing from the wrong framebuffer at one point, missing out on a necessary 32->16 reinterpret (fixed now). Also framebuffers cloned for reinterpret were too small, fixing that gets us to:

[hrydgard]

One trivial bug fix later:

Only the stripes left, maybe some precision issue, it's kinda weird.

Seems to be framebuffers getting resized to weird mismatching dimensions, then reinterpret going very slightly wrong. During reinterpret, pixel exactness is needed...

[hrydgard]

Getting closer, 2x:

1x:

Note how there's weird yellow fringes in the sky.

Anyway, that might be good enough to get it in, for further improvement afterwards...