Software: Fix zfreeze with CullMode::All

[Pokechu22]

CullMode::All is often used with zfreeze to draw a triangle for the reference plane without it showing up (see the note on GX_SetCoPlanar). However, this was broken in the software renderer. I've fixed it by re-using the infrastructure in VertexManagerBase used by the hardware backends, but this is a bit jank.

[Pokechu22]

It seems like there was a second issue: the slope used with zfreeze should use offsets from the screen's origin, but instead it was using offsets from the triangle's origin. This is what caused the triangles in the sky in RS2: half of them had the first vertex at the top of the triangle, and the other half had the first vertex at the bottom of the triangle.

[phire]

but this is a bit jank.

Indeed. A better fix is to disable the code in vertex manager which filters out CULL_ALL draw calls and forward them to video software. And then filter them out per triable, in the Rasterizer::Draw after the z slope has been calculated.

[Pokechu22]

The equivalent code in the hardware backends looks like this:

dolphin/Source/Core/VideoCommon/VertexManagerBase.cpp

Lines 574 to 592 in e92e90d

	float dx31 = out[8] - out[0];
	float dx12 = out[0] - out[4];
	float dy12 = out[1] - out[5];
	float dy31 = out[9] - out[1];
	float DF31 = out[10] - out[2];
	float DF21 = out[6] - out[2];
	float a = DF31 * -dy12 - DF21 * dy31;
	float b = dx31 * DF21 + dx12 * DF31;
	float c = -dx12 * dy31 - dx31 * -dy12;
	// Sometimes we process de-generate triangles. Stop any divide by zeros
	if (c == 0)
	return;
	m_zslope.dfdx = -a / c;
	m_zslope.dfdy = -b / c;
	m_zslope.f0 = out[2] - (out[0] * m_zslope.dfdx + out[1] * m_zslope.dfdy);
	m_zslope.dirty = true;

Apart from some of the excessive minus signs and other confusion that also originally existed in InitSlope, there's a divide by zero check that doesn't exist in the software renderer. Does this actually matter?

The check was added in commit add59b3, but #1812 (comment) says it shouldn't happen with zfreeze specifically.

[phire]

I think my comment is saying this based on the fact that games that use zfreeze should be sane enough to use non-degenerate triangles for their z reference planes.

No idea what happens on real hardware. These are zero-sized triangles that won't even get rasertized, right? Which is why they don't have a valid slope. It's entirely possible they have already been discarded by this point in the pipeline.

[dolphin-emu-bot]

FifoCI detected that this change impacts graphical rendering. Here are the behavior differences detected by the system:

• djfny-menu on sw-lin-mesa: diff
• mtennis-zfreeze on sw-lin-mesa: diff
• rs2-skybox on sw-lin-mesa: diff
• rs2-zfreeze on sw-lin-mesa: diff
• rs3-bumpmapping on sw-lin-mesa: diff
• rs3-skybox2 on sw-lin-mesa: diff

_{^{automated-fifoci-reporter}}