VideoCommon: update TextureCache logic for finding oversized XFBs

[iwubcode]

Previously, when looking for XFBs in the texture cache, if their dimensions didn't match VI we were recomputing the hash. However, when we did so we weren't using the same hash logic as we did when actually hashing the entry.

Additionally, when dealing with oversized XFBs (which are resized on creation) the hashing algorithm is actually different than the one used to hash the VI registers that we compare against. This causes these copies to be invalidated, meaning we will fall back to stitching from guest memory (if it exists).

This change fixes both of those issues. This seems to resolve 12392

[iwubcode]

@stenzek - curious if you have any thoughts on this one!

[phire]

Also, looking though texture cache code. I find it a little suspicious that we have separate stitching functions for xfb and efb textures (StitchXFBCopy and DoPartialTextureUpdates). It's a bad sign that neither function is generic enough.

The original intention of merging the xfb cache into texture cache was so that only one function to stitch copies would be needed. That it would be generic and able to handle every single usecase.

There is no fundamental difference between efb and xfb copies. They just have a different texture format.

[phire]

Looks like I left a long comment on this topic back when the separate path was originally PRed: #7412 (comment)

Fixing the fundamental issue (that dolphin's texture cache was implemented at the level of whole textures and any kind of handling for non-matching transformations has been hacked on) probably requires major refactors.

[iwubcode]

Also, looking though texture cache code. I find it a little suspicious that we have separate stitching functions for xfb and efb textures (StitchXFBCopy and DoPartialTextureUpdates). It's a bad sign that neither function is generic enough.

Yeah, when originally doing HybridXFB, there was a reason I couldn't consolidate the two completely. Well and DoPartialTextureUpdates is a horrible name :P.

I believe the difference was in the NES zelda in Zelda Collector's Edition. Something about there being some bounds difference? Maybe there have been enough changes to warrant another look.

Looks like I left a long comment on this topic back when the separate path was originally PRed

I remember that. I agree, that would get rid of all the weird EFB specific checks we do right now. All the mess in GetTexture. Though, I think we'd have more work to do to handle hires textures.

I'd be interested in looking into that. Maybe once all my enhancement features are finished.

[phire]

and DoPartialTextureUpdates is a horrible name :P.

Because when it was originally created, it did one thing only. Handle partial texture updates to the main texture atlas in NSMB Wii.

It's been expanded and has grown into a monster since then.

[iwubcode]

@JMC47 - please retest when you have time!

[JMC47]

This no longer fixes the games in the latest revision.

[JMC47]

It is now fixing the games again.

[JMC47]

Let's get this merged early in the month in case there are some unknown edge cases beyond what I'm aware of that crop up.