Graphical glitch with two directional lights and ViewportContainer (background visible with inverted colors)

[Demindiro]

I don't know the term for this specific glitch, so if someone does know please amend the title.

Possibly related: #43760

Godot version:

3.2.3.stable.custom_build.662455ee8

OS/device including version:

Linux pc 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux

GeForce GTX 1060 6GB
NVIDIA-SMI 418.152.00   Driver Version: 418.152.00   CUDA Version: N/A 

Issue description:
If you have a mesh in a ViewportContainer and two (directional?) lights, the background is visible through the mesh with inverted colors. The strength of the effect seems to depend on the angle of a face to the camera. This does not happen when using GLES2.

Steps to reproduce:

• Create a project with GLES3
• Add a Camera with Cull Mask set to only layer 1.
• Add a ViewportContainer
  • Add a Viewport as a child with Transparent Bg enabled.
    • Add a Camera with Cull Mask set to only layer 2.
      • Add a MeshInstance with any mesh and Layer Mask set to layer 2 (and ensure it is visible for the second camera).
• Add two DirectionalLights. Their orientation doesn't matter.
• (Optional) Add a MeshInstance with any mesh and Layer Mask set to layer 1

Minimal reproduction project:

viewport_inverted_bg_color.zip

[lyuma]

I'll just go out on a limb and say that this is expected behavior. Please clamp your alpha before blending. Godot is lacking a bit of functionality to make this easy but it's possible to workaround.

So let me go back and explain what's going on here.
Because you have two directional lights, and Godot 3.2 only supports one directional light per render pass, you're rendering your object twice, once with GL_ONE GL_ZERO blend function, and again with GL_ONE GL_ONE blend function. What this means, is you are taking the base color of your object with one directional light, and adding the color of the second directional light contribution. This is all fine for color. BUT, in a naive implementation, the same blend function is used for alpha. This means you take the base alpha (1.0) and add it with the second directional light alpha (1.0). This produces an alpha of 2.0

Why is alpha of 2.0 bad? Well, a naive ViewportTexture shader will look like this:

void fragment() {
    COLOR = texture(TEXTURE, UV);
}

By default, Godot CanvasItem shaders use a blend function of GL_SRC_ALPHA GL_ONE_MINUS_SRC_ALPHA
What this means is you will take your source alpha (2.0) and do the following:
blended color = 2.0 * SRC_COLOR + (1.0 - 2.0) * DST_COLOR
where SRC_COLOR is what you're rendering, and DST_COLOR is what was painted on the screen before.

The math then literally works out to 2 * SRC_COLOR - DST_COLOR, so invert what is on the screen and paint an oversaturated version of your render texture on top, which is exactly what your screenshot shows.

One "fix" is simple. Edit the shader for your viewport texture and modify the shader to the following:

void fragment() {
    COLOR = texture(TEXTURE, UV);
    COLOR.a = clamp(COLOR.a, 0.0, 1.0);
}

Clamping alpha will prevent undesired blending behavior.

Oh, one thing I'll mention. Disabling HDR on your viewport texture (by setting it to RGBA_8 pixel format) should inherently clamp the alpha. I'd suggest considering that as a simple workaround. I cannot for the life of me figure out how to do this in Godot, so I'll leave it as an exercise to the reader. But basically, if the texture cannot represent alpha outside the range of 0.0 to 1.0, then you're never going to end up with an alpha of 2.0 and cause this problem in the first place.

Now, this might be more performant, but this technically might not be the best looking way to fix the issue, and in fact is likely to break down when transparent objects are rendered. To have better control, Godot needs to be improved so that we can directly control the blend functions and blend equations, or it needs to internally perform mitigations to this issue when used as a ViewportTexture (as tends to be the godot way).

(Also, just so you know, this is all only an issue because you have multiple directional lights. That's the only thing Godot 3.2's GLES3 renderer is unable to handle. For performance reasons, you'd do better by having the second light work be a spot or point light.)

Anyway, I'll explain some of what you would do in a pure OpenGL ES program, but note that this is all moot because Godot 3.2 is already released and unlikely to completely rework blending, and Godot 4 will use Vulkan, and additionally it renders all lighting in one pass (in 3D)... though maybe this still matters in 2D or in the low end renderer:

Basically, you can deal with this by using separate blend functions for the alpha channel:
https://www.khronos.org/registry/OpenGL-Refpages/es3.0/html/glBlendFuncSeparate.xhtml
https://www.khronos.org/registry/OpenGL-Refpages/es3.0/html/glBlendEquationSeparate.xhtml

Is a normal OpenGL ES program, you can clamp alpha by painting a square over the screen, blending color with ZERO ONE and blending alpha with ONE ONE and blend equation GL_MIN:
This should produce the same result as our shader hack above, and would be slower as it is forced to do another fullscreen blit pass. I just wanted to mention it because maybe you cannot control the shader but still want a clamped alpha in your render texture.

What I would suggest for additive lighting passes is using glBlendFuncSeparate and using alpha equation of GL_ONE GL_ZERO or GL_ZERO GL_ONE. I think for transparent objects, you might want to do something slightly different, not sure.