[F3D] Combiner Overflow Preview Fix 2

[jesusyoshi54]

The Shade input on C was causing overflow because it was registering as slightly above one due to float imprecision. To create a proper distance between 255 inputs vs 256 inputs, I made the 1.0 value 1.0001 and the overflow now occurs at 1.0001. This should make it so that only Combined values can overflow, as they're the only ones that can be >255 on the C input (besides for YUV K5 but fast64 does not allow K5 values above 1.0).
Fixes #348

[Dragorn421]

Why

made the 1.0 value 1.0001

?
(you are speaking of the 1 input to the combiner right?)

it sounds like that just makes it more likely for 1 to wrongly wrap to -1 on the C combiner inputs

[Dragorn421]

Testing wise, unlike the previous attempt OoT's imported kokiri forest and deku tree are fine :)

[jesusyoshi54]

Why

made the 1.0 value 1.0001

? (you are speaking of the 1 input to the combiner right?)

it sounds like that just makes it more likely for 1 to wrongly wrap to -1 on the C combiner inputs

It makes it so that it correctly wraps with an input of 1.0 on C. There is a reason why C by default doesn't allow an input of 1.0 but every other input does. 1.0001 represents 256 in a float number system and we need it to actually wrap on 1.0001 to get the weird CC behavior or else it won't mirror the actual CC. Here is a test image, with 1 cycle passing 1.0, and the top left is shade*combined, and the top right is combined*shade. The thing that is causing the issue here is the overflow of 1.0, which makes it overflow once shade reaches a value less than 0x80. I can't do anything about the way vertex colors interpolate, but other than that this matches.

Here is a code example to make it more obvious, the equations in python were shared by Tharo when this was first discussed.

def ext(x):
    return x | -(x & 0x100)

def spext(x):
    return (x | ~0x1FF) if ((x & 0x180) == 0x180) else (x & 0x1FF)

def cc_calc(a,b,c,d):
    a = spext(a)
    b = spext(b)
    c = ext(c)
    d = spext(d) << 8
    combined = (a - b) * c + d
    return (combined + 0x80) & 0x1FFFF

and then the test case. Overflow occurs at 384 (1.5) on the final output.

# first cycle examples
>>> cc_calc(0, 0, 0, 256) # (0 -0) *0 + 1.0
65664 # 256
>>> cc_calc(256, 255, 255, 255) # (1 - prim) * tex0 + prim
65663 # 256

# now with C input, we can see that we get overflow with A inputs < 0.5
>>> cc_calc(0x80, 0, 65663>>8, 0)>>8
384
>>> cc_calc(0x81, 0, 65663>>8, 0)>>8
383
>>> cc_calc(0x7F, 0, 65663>>8, 0)>>8
385
# and as a comparison, here is the same test with combined on A
>>> cc_calc(65663>>8, 0, 0x7f, 0)>>8
127
>>> cc_calc(65663>>8, 0, 0x80, 0)>>8
128
>>> cc_calc(65663>>8, 0, 0x81, 0)>>8
129

So yeah, 1.0 exactly on C is doing weird stuff. Here is an image with 1.0 passed

[Dragorn421]

I think I get it, didn't know about this, that's weird
thanks for the details