Allow depth-writing shaders to work with shadow maps

[Zylann]

Describe the project you are working on

A node rendering SDF models with raymarching: https://github.com/Zylann/godot_sdf_blender

Describe the problem or limitation you are having in your project

The node cannot cast shadows, and cannot receive shadows.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

In my use case, the node uses a simple geometry primitive (such as a quad, a box, or a sphere). The goal of that primitive is solely to render pixels in a certain location on screen. It might be displaced to be fullscreen (like post-processing effects do by assigning POSITION), or only part of the screen when a box is chosen.

But unlike post-processing effects, DEPTH is written into, as well as NORMAL, ALBEDO and other various pixel properties, and discard is used to cut out the edges of the model. This allows the result to fit nicely in a scene with regular meshes.
The AABB is also altered to match the real extents of the object.

Because the shader writes to DEPTH, it provides the information for shadowmaps to also work on it. Currently that doesn't seem to happen.
I tried using a cube without displacing it and the resulting shadow was following the cube and not the real depths.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Take DEPTH-writing shaders into account?
I'm not expert in that area, but from what I understand of shadow maps, they render depth from the point of view of the light, and the result is used by the regular render pass to darken occluded pixels. So it sounds like it should be possible.
If it incurs at the expense of other "classic" shaders, maybe add a render_mode to treat the shader differently? Or have a shadow() function to allow customizing the shadow pass?

This should ideally not take effect on shaders that actually just do post-processing, because while those displace their quad to be fullscreen, it doesnt mean such quad should put everything in shadow by being right in front of the frustrum. Perhaps it would just work as long as they dont write to DEPTH.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No. Unless you re-implement a shadow mapping system on top of the VisualServer API using viewports, it's going to be terribly inefficient, hard to maintain and share as a plugin.

Is there a reason why this should be core and not an add-on in the asset library?

It targets a part of the rendering engine.

---

NB: this is an area I only started experimenting, so maybe Godot actually supports this and I got something wrong somewhere. If that's the case this proposal could be closed, but at the moment it's not certain yet ;)

[Calinou]

See also #1843 (comment).

[drcd1]

In 3.2.3-stable (at least), depth-writing shaders already write to shadow maps (and to any depth pre-pass, in general) if they use alpha-cutoff or discard;. If understand the code correctly, using any of these means they run the whole fragment shader when doing depth pre-pass (like when rendering shadow maps). They can cast shadows, they just can't receive them.

I would suggest checking if depth is written to, in rasterizer_scene_gles3.cpp, in the same if statement where uses_discard is checked.

For now, a work around would be to just do something like "if(false) discard;" in the shader, to trigger p_material->shader->spatial.uses_discard to be true.

Note: there would still be some problems with the shadow maps, with bias or with dual paraboloid maps, but that would require further thought.

See also #974 (comment)

[Zylann]

depth-writing shaders already write to shadow maps (and to any depth pre-pass, in general) if they use alpha-cutoff or discard;

The shader shown in the screenshot uses discard already: https://github.com/Zylann/godot_sdf_blender/blob/master/addons/zylann.sdf_blender/raymarch.shader

[drcd1]

That's because SCREEN_UV doesn't work for the shadow passes, I think.
I changed your code (by creating a varying and setting to be the same as VERTEX.xy, and using that instead of SCREEN_UV in fragment) and shadows are cast, but they are not received correctly by the sdf.

[viktor-ferenczi]

Ran into this issue with a depth writing shader. The resulting (carved out) surface receives some shadow, but it corresponds to the original mesh (a cube), not to the 3D surface defined by the DEPTH written by the shader. It drops proper shadow according to its carved out shape. It cannot shadow itself.

It would be nice to have a way to fix the shadows in this case, even at a performance cost.

[viktor-ferenczi]

I think the fragment shader may just allow overwriting the VERTEX variable, so it can correct the view space position according to the DEPTH applied.

In the shader code it should override vertex_interp, so the correct position is used for the lighting calculations, not just the interpolated vertex positions. (If I interpret the shader code well in Godot's sources.)

[viktor-ferenczi]

PR-65307

Implemented the above change. It works perfectly, at least on Windows with Vulkan. Objects with a DEPTH setting fragment shader now receive shadow and even cast shadow on themselves.

I'm aware that it may not be ideal to make VERTEX writable, but it perfectly solves the problem. We may want to introduce an out only variable instead, but it may result in slightly less performant shader code.

The PR contains a FIXME comment on a performance optimization to my change. I don't know how to make that code block conditional. It should be added only if the game's fragment shader actually writes to the VERTEX variable. Could a Godot dev with experience in shader implementation please help in this? (10 days ago I did not even know how to write shaders...)

The documentation update will also follow once we agree in a final solution.

Test object: Cube mesh containing the whole object. The fragment shader carves this shape out.
This same fix should also work for SDF objects, but that would need setting the VERTEX in its fragment shader.

Original bogus shadows, notice the lack of shelf-shadow and that the ball's shadow is casted on the container cube:

Correct shadows with the Godot PR and writing correct VERTEX view coordinates in the shader:

[viktor-ferenczi]

Spin-off proposal