Outrun 2006: Car reflections are missing

[thedax]

A continuation of #5506 so we don't get lost or confused with old comments, as the colour issue is fixed.

Summary of the issue: Cars should have a glossy and shiny look to reflect light and the sky, but they don't, and are instead dull and bland looking. This issue persists into gameplay as well; it's not exclusive to menus.

Here are two screenshots, the first is Vulkan HW, and the second is Vulkan SW.
Vulkan:

Software (Vulkan backend):

It's not that easy to tell, but the SW picture does indeed have the glossy look. I also included two GE debugger dumps, one from each renderer, to showcase the issue.

DUMP.zip

It's also worth noting all four backends are affected by this, and only software rendering displays this effect correctly, currently. Also, in the past, the cars did have the glossy look, but it was incorrect, causing every car to have the wrong colour (hence why this is a continuation of that old issue).

[thedax]

I did some sleuthing, and here's roughly where each renderer lost the (incorrect/wrong coloured) reflections, testing between versions 1.0 through 1.6:

OpenGL: Displays in 1.0 stable, but not 1.1 stable and beyond
Direct3D9: Loses its reflections in version 1.5 and beyond, 1.4 stable and below display them
Vulkan: Has never displayed the reflections, to my knowledge (and didn't exist before version 1.3)
Direct3D11: Has never displayed the reflections, to my knowledge (and didn't exist before version 1.4)

OpenGL 1.0 -> 1.1: v1.0...v1.1.0
D3D9 1.4 -> 1.5: v1.4...v1.5

[shpuld]

The bug might also be related to the broken water reflections in the same game, some pools of water don't show reflections at all without software rendering. If you step through the frames in graphics debugger you can see how it seems to render a cube map and a sphere map to textures but you won't see them applied

[hrydgard]

The game does a lot of difficult-to-emulate graphics trickery, it would not surprise me if the causes for this are similar to the missing water, see #11928

[hrydgard]

OK so I've figured out what's going on here (though investigate in gameplay, not at this screen). It's successively building up an environment map, by rendering the sides of a cube map to a 565 texture at 0x040d4000, then mesh-distorting each built side into a bigger format 4444 texture at 0x040cc000 which becomes the environment map. When drawing the car it textures from 0x040cc000 using 4444, which should be all good - it uses alpha from that texture to blend it.

However! Right before texturing from it, it switches to 565 rendering and draws a gradient into the blue channel, with an RGB color mask of 0xF00000. This flags the texture as 565 so it will not be matched for texture-from-framebuffer in the final bit. I first could not understand this but it has a logical explanation. That mask, in 565 mode, means rendering to 0xF000 which is exactly alpha in 4444 mode! The PSP cannot draw color to the alpha/stencil channel, it can only blend, do stencil ops and clear, that's why the game is hackily using 565 mode to draw a gradient into the alpha channel of a 4444 texture. GAH.

I have a plan for how to emulate this correctly, but it'll take some doing. At each of these transitions, we'll simply have to reinterpret the colors into the new format. We'll also have to reinterpret again when texturing. A few more tiny render passes...

[hrydgard]

Final part of the fix is here: #13640

[hrydgard]

Fix merged!

(Still doesn't work on D3D9 and OpenGL 2.0. Most ES 2.0 devices aren't fast enough for the game anyway so...)