[F3D] Material Overflow Preview Fix

[jesusyoshi54]

Redo of #313

Fixed the way overflow works on the color combiner.

• Previous behavior was:
  • overflow [-0.5, 1.5) per cycle combined
• New behavior is:
  • c overflows [-1.0, 1.0) on input
  • overflow [-0.5, 1.5) per cycle combined

On 2 cycle, the combiner doesn't actually do the special sign extension [-0.5, 1.5) on the first cycle's output unless it is for alpha compare, but since this will feed into inputs which are getting special sign extension on hardware it replicates the behavior properly.

and a response from the prev PR

to be clear,

overflow [-0.5, 1.5)

means "clamping into the [0, 1] range happens for values in range [-0.5, 1.5), values outside are not clamped properly" right? (not sure on the specifics of "not clamped properly")

Values are coming out of the combiner have a special sign extension and clamp happening at those values, you can see that from AL source here with the definition of the clamp here.

So this means the output has values clamped to [0-1), with the overflow happening on [-0.5, 1.5). Anything [1-1.5) = 1, anything [-0.5-0] = 0.

On 2 cycle, the combiner doesn't actually do the special sign extension [-0.5, 1.5) on the first cycle's output unless it is for alpha compare, but since this will feed into inputs which are getting special sign extension on hardware it replicates the behavior properly.

can you clarify what is fast64 nodes behavior and what is hardware behavior here

I'll just go over this in steps. First the actual N64:

• Inputs receive the sign extensions they're assigned. e.g. A, B, & D get special sign extension, C gets normal one
• First cycle goes through the normal equation, gets right shifted by 8 (so that it is a s1.15 number).
• If there is no second cycle, this gets the special clamping described above, otherwise the number goes straight to cycle 2.
• Second cycle's inputs get the sign extensions assigned.
• Second cycle's output gets the special clamping.

For Fast64 both cycles use the same nodes, so the formula is as follows:

• C input receives normal sign extension
• Combiner equation is calculated
• Output gets special sign extension
• Value is clamped [0-1) heading into blender (which is just fog/no fog for fast64).

[Lilaa3]

I've seen this behavior before from constantly playing with overflow and underflow, it's nice to see an explanation and the fix aha.
The nodes are doing exactly what you described, so I will approve, I will let merging left to dragon since he discussed the pr with you first

[Dragorn421]

Nice 👍
Lilaa approved some time ago so merging now