ShaderGen: Always calculate lighting for both color channels #4601

stenzek · 2017-01-03T12:57:39Z

Up until now, we assumed that the XFMEM_SETNUMCHAN register controlled how many channels the lighting pipeline processes. If this was set to zero, no lighting was calculated at all, and the colors were passed through as-is from the corresponding attribute.

Cel-Damage is an interesting case, as it doesn't module the vertex color with the texture result as you would expect. Instead, it uses two sets of texture coordinates. The first reference a diffuse texture, as usual. The second set of texture coordinates, however, are derived from the vertex color. This set indexes a LUT-like texture, which is blended with the first texture result in a second TEV stage, to produce the lit pixels.

As we were setting the vertex color to white before calculating texture coordinates, this resulted in the same texel (at normalized coordinates 1,1) being used from the LUT texture, and flat, brightly-colored polygons. The software rasterizer used black as a "default" color, resulting in flat, dark-colored polygons.

This PR changes the shader generation to always process the lighting pipeline, and only "clear" the colors based on XFMEM_SETNUMCHAN after texture coordinate generation, fixing the lighting in Cel-Damage.

I've tested this behavior on hardware by using the lighting pipeline's color as texture coordinates, as cel-damage does, and comparing results when XFMEM_SETNUMCHAN is set to zero and one. No change is visible, confirming that the lighting pipeline does indeed run regardless of this register's value.

This opens up the door to further questions though, see my comment below for more info. However, to keep things simple, this change is self-contained, fixes cel-damage, and has no other regressions in fifoci.

JMC47 · 2017-01-04T06:19:10Z

Fixes Cel-Damage. Is there anything else I should test?

Source/Core/VideoBackends/Software/TransformUnit.cpp

@@ -319,7 +319,7 @@ static void LightAlpha(const Vec3& pos, const Vec3& normal, u8 lightNum, const L

 void TransformColor(const InputVertexData* src, OutputVertexData* dst)
 {
-  for (u32 chan = 0; chan < xfmem.numChan.numColorChans; chan++)
+  for (u32 chan = 0; chan < 2; chan++)


Source/Core/VideoCommon/LightingShaderGen.cpp

  {
    object.Write("{\n");

    bool colormatsource = !!(uid_data.matsource & (1 << j));
    if (colormatsource)  // from vertex
    {
-      if (components & (VB_HAS_COL0 << j))
+      if (j < numColorChans && components & (VB_HAS_COL0 << j))


stenzek · 2017-01-04T14:05:09Z

Marking as WIP until I get a chance to figure out the regression in Mario Sunshine.

JMC47 · 2017-01-07T19:03:24Z

Vegas Party Depth is the only real regression here. The Brawl characters are already incorrect.

stenzek · 2017-01-31T05:49:47Z

This should be regression free now. Also fixes the grey boxes in Super Mario Sunshine on the hardware backends, and Tony Hawk on the software backend. I'm still not sure about the default values, it would be nice to verify this with a hardware test, but I've spent way too much time on this PR already.

Furthermore, I suspect we should be using the number of elements field in the vertex format (at least for the color channels) when processing the vertices, but this is a larger change, and as I mentioned in the comment the way I've done it here catches the known cases for now.

JMC47 · 2017-01-31T15:14:11Z

LGTM. Should we merge this for this progress report or wait for the next?

JMC47 · 2017-04-04T07:15:24Z

Can this be rebased? I'd like to test it on something.

Source/Core/VideoCommon/VertexShaderGen.cpp

@@ -23,7 +23,6 @@ VertexShaderUid GetVertexShaderUid()
  memset(uid_data, 0, sizeof(*uid_data));

  _assert_(bpmem.genMode.numtexgens == xfmem.numTexGen.numTexGens);
-  _assert_(bpmem.genMode.numcolchans == xfmem.numChan.numColorChans);


shuffle2 · 2017-06-06T07:45:46Z

ping? :>

iwubcode · 2017-06-20T03:00:23Z

I ran this through fifo_comparer against the fifoci tests (using current master as a base) and the only issue was the smsGreyCubes2.dff (the cubes are missing) which occurs on all backends. If this can be rebased, I can try it again.

JosJuice · 2017-06-20T06:14:35Z

@iwubcode Not rendering the gray cubes in Super Mario Sunshine is actually console-accurate.

JMC47 · 2017-06-20T06:16:17Z

This only fixes things according to fifoci

…

On Tue, Jun 20, 2017, 2:14 AM JosJuice ***@***.***> wrote: @iwubcode <https://github.com/iwubcode> Not rendering the gray cubes in Super Mario Sunshine is actually console-accurate. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4601 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGSuQUc6gsffPbfJqYN8QkcpHMq-LWkuks5sF2NWgaJpZM4LZmGT> .

Source/Core/VideoBackends/Software/SWVertexLoader.cpp

+                                 const PortableVertexDeclaration& vdec)
+{
+  bool use_color_1 = vdec.colors[0].enable && vdec.colors[1].enable;
+  for (size_t i = 0; i < 2; i++)


nbohr1more · 2017-09-10T18:39:19Z

Rebase?

stenzek · 2017-11-21T15:13:14Z

I've decided to split this PR into two, one to fix SMS and one to fix cel-damage. The SMS fix will come later, after additional hardware tests, but this PR (cel-damage fix) is good to go now.

Confirmed on hardware that the lighting pipeline still occurs even when numColorChans is zero, and these colours are used for texture coordinate generation. If numColorChans is zero, these values are not sent from the vertex stage to the rasterizer.

This is what the PR modifies, the behavior below is unchanged and left for a future PR.

The next question:
What are the values of the colour inputs to TEV when numColorChans is less than the used colour index? Hardware tests show that the value is whatever last passed through the pipeline, which is painful to emulate (i.e. it's a register in hardware, and the register is not updated if numColorChans is less than its index).

But do we need to emulate the exact hardware behavior here? Or can we get away with just setting the colour inputs to zero.

Source/Core/VideoCommon/UberShaderCommon.cpp

-  out.Write("%sfor (uint chan = 0u; chan < xfmem_numColorChans; chan++) {\n",
-            api_type == APIType::D3D ? "[loop] " : "");
+  out.Write("%sfor (uint chan = 0u; chan < %uu; chan++) {\n",
+            api_type == APIType::D3D ? "[loop] " : "", static_cast<u32>(NUM_XF_COLOR_CHANNELS));


Cel-damage uses the color from the lighting stage of the vertex pipeline as texture coordinates, but sets numColorChans to zero. We now calculate the colors in all cases, but override the color before writing it from the vertex shader if numColorChans is set to a lower value.

Fix Super Mario Sunshine debug cubes (originally PR #4601 by stenzek)

Fixes https://bugs.dolphin-emu.org/issues/12620 The changed code did not match the corresponding code in VertexShaderGen. Some parts of the sky have 2 color channels in each vertex, while others only have 1, despite only color channel 0 being used and XFMEM_SETNUMCHAN being set to 1 for both of them. The old code (from dolphin-emu#4601) caused channel 0 to be set to channel 1 if the vertex contained both color channels but the number of channels was set to 1, which is wrong.

stenzek changed the title ~~FifoCI test run~~ ShaderGen: Calculate lighting for all color channels Jan 3, 2017

stenzek added the WIP / do not merge Work in progress (do not merge) label Jan 3, 2017

stenzek changed the title ~~ShaderGen: Calculate lighting for all color channels~~ ShaderGen: Always calculate lighting for both color channels Jan 4, 2017

stenzek removed the WIP / do not merge Work in progress (do not merge) label Jan 4, 2017

BhaaLseN reviewed Jan 4, 2017

View reviewed changes

degasus reviewed Jan 4, 2017

View reviewed changes

stenzek added the WIP / do not merge Work in progress (do not merge) label Jan 4, 2017

stenzek removed the WIP / do not merge Work in progress (do not merge) label Jan 31, 2017

ExE-Boss reviewed Apr 8, 2017

View reviewed changes

iwubcode reviewed Jun 21, 2017

View reviewed changes

Source/Core/VideoBackends/Software/SWVertexLoader.cpp Outdated

const PortableVertexDeclaration& vdec)

{

bool use_color_1 = vdec.colors[0].enable && vdec.colors[1].enable;

for (size_t i = 0; i < 2; i++)

This comment was marked as off-topic.

Sign in to view

VideoCommon: Use constant for number of color channels in XFMemory

18c1bf1

degasus reviewed Nov 21, 2017

View reviewed changes

Helios747 merged commit 066471b into dolphin-emu:master Nov 22, 2017

stenzek deleted the celdamage-lighting branch February 19, 2018 15:35

Pokechu22 mentioned this pull request Sep 30, 2020

Fix Super Mario Sunshine debug cubes (originally PR #4601 by stenzek) #9122

Merged

leoetlino added a commit that referenced this pull request Nov 21, 2020

Merge pull request #9122 from Pokechu22/pr-4601-test

608dd6a

Fix Super Mario Sunshine debug cubes (originally PR #4601 by stenzek)

Pokechu22 mentioned this pull request Aug 9, 2021

UberShaderVertex: Fix Tony Hawk Pro Skater 4 #10012

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ShaderGen: Always calculate lighting for both color channels #4601

ShaderGen: Always calculate lighting for both color channels #4601

stenzek commented Jan 3, 2017 •

edited

Loading

JMC47 commented Jan 4, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

stenzek commented Jan 4, 2017

JMC47 commented Jan 7, 2017

stenzek commented Jan 31, 2017 •

edited

Loading

JMC47 commented Jan 31, 2017

JMC47 commented Apr 4, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

shuffle2 commented Jun 6, 2017

iwubcode commented Jun 20, 2017 •

edited

Loading

JosJuice commented Jun 20, 2017

JMC47 commented Jun 20, 2017 via email

This comment was marked as off-topic.

nbohr1more commented Sep 10, 2017

stenzek commented Nov 21, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

ShaderGen: Always calculate lighting for both color channels #4601

ShaderGen: Always calculate lighting for both color channels #4601

Conversation

stenzek commented Jan 3, 2017 • edited Loading

JMC47 commented Jan 4, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

stenzek commented Jan 4, 2017

JMC47 commented Jan 7, 2017

stenzek commented Jan 31, 2017 • edited Loading

JMC47 commented Jan 31, 2017

JMC47 commented Apr 4, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

shuffle2 commented Jun 6, 2017

iwubcode commented Jun 20, 2017 • edited Loading

JosJuice commented Jun 20, 2017

JMC47 commented Jun 20, 2017 via email

This comment was marked as off-topic.

nbohr1more commented Sep 10, 2017

stenzek commented Nov 21, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

stenzek commented Jan 3, 2017 •

edited

Loading

stenzek commented Jan 31, 2017 •

edited

Loading

iwubcode commented Jun 20, 2017 •

edited

Loading