VideoCommon: copy software renderer logic for blend mode priorities

[Tilka]

I've not tested this on hardware, but it fixes issue 12271 (shadow people in Deal or No Deal - Special Edition).

The software renderer uses (pseudo code):

  if (bpmem.blendmode.blendenable)
  {
    if (bpmem.blendmode.subtract)
      SubtractBlend
    else
      BlendColor
  }
  else if (bpmem.blendmode.logicopenable)
    LogicBlend
  else
    dstClrPtr = color;

Until now VideoCommon used:

  if (bpmem.blendmode.subtract)
    SubtractBlend
  else if (bpmem.blendmode.blendenable)
    BlendColor
  else if (bpmem.blendmode.logicopenable)
    LogicBlend
  else
    dstClrPtr = color;

The diff is much smaller when whitespace is ignored.

[Tilka]

Before:

After:

[Pokechu22]

This seems like a plausible fix. It looks like OpenGL handled it this way prior to 90254bd (March 2009), and that commit doesn't seem to be specifically targeting this behavior. (The authoritative-sounding comment was added in 3df8284/#4581.) The software renderer has always had it, ever since it was added in 9b16c36 (October 2009 - odd that this is after the other commit).

Hardware testing of some sort would be nice, but probably isn't needed.

[iwubcode]

Code change matches sw renderer and LGTM. Tested with a handful of games/personal fifologs.

[JosJuice]

Has a FIFO log for this been added to FifoCI?

[Tilka]

Has a FIFO log for this been added to FifoCI?

No, how do I do that?

[Pokechu22]

I think @JMC47 needs to add them (it's part of an admin interface of some sort), which requires both a fifolog (DOND_SE_Blend_Issue_Career.zip from https://bugs.dolphin-emu.org/issues/12271) and a description for it (something like "Deal or No Deal - Special Edition renders its character models with blending disabled but with the blending subtract bit set. If subtraction blending is actually used, the scene renders incorrectly.")