Add an overlay material property to GeometryInstance rendering on top of active materials for the entire geometry

[fbcosentino]

Describe the project you are working on

Currently working on a VTuber app (animates 3D character based on motion capture data) with transition effects applied on user-provided .glb or .vrm models (which I therefore have no control on, or knowledge of, beforehand).

Previous projects which would be much improved by this feature were:

• a 2.5d platformer (implemented as 3d with locked Z axis) with several visual effects
• a point-and-click app using highlight materials for object picking
• 2 games using toon shading, where this feature would simplify the workflow around outline shader

Describe the problem or limitation you are having in your project

Applying overlay materials (such as momentary visual effects) on multi-surface meshes currently requires adding a next pass material to all the surfaces, which might be cumbersome when the material is to be applied to a range of different geometries. E. g. an enemy plays a VFX when receiving a blaster hit, but each enemy type is a different mesh, with different number of surfaces.

This also makes it not very trivial to use AnimationPlayer to control the material. In the previous example, controlling the effect via animations requires either adding a different AnimationPlayer to each different type of mesh, with manually set animations for each of the surfaces (which means only prepared targets could show the effect), or - if a same AnimationPlayer-based scene is instanced in any of the affected enemies - a script iterating over the surfaces of the mesh assigning the material, and exposing to the AnimationPlayer somehow (e.g. via export var).

The material_override property is not an option as it works replacing the active material for the surfaces, not adding a new pass.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This proposal is to add a material_overlay property to GeometryInstance (and therefore MeshInstance), which is a Material affecting all surfaces like material_override does, but rendering as a new pass on top of the active materials, instead of replacing them.

Example

Test model: mesh with 3 surfaces, each with a different material

Test effect added as next pass in first surface - provides a single point of access for an AnimationPlayer but needs scripting to be assigned to other surfaces - or repetition in the animation itself.

Test effect added as material_override - direct interface with single material, but underlying materials are lost

Test effect using material_overlay - effect covers entire model preserving base materials and providing a single access for animations

GIF below demonstrates using Next pass to stack 3 material effects in material_overlay, all controlled via a single animation in AnimationPlayer. This AnimationPlayer can be saved as branch scene and instanced in any affected MeshInstance during runtime as needed. The only scripting required is assigning the material to material_overlay, which is completely independent from mesh materials or number of surfaces.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Implemented in the PR 50823: godotengine/godot#50823

This PR adds the property to GeometryInstance linked to the InstanceBase.material_overlay field in VisualServer, and adds a new call to _add_geometry_with_material in the rasterizer for the overlay material (and corresponding next passes) after the active material.

The PR implements in both GLES2 and GLES3 for 3.x but discussion is needed on how to implement in master, if the feature is approved.

If this enhancement will not be used often, can it be worked around with a few lines of script?

As proposed, no. In some cases the visual results can be achieved by adding the material as second pass to every surface in the mesh and manually adding several animation tracks in every model where the effect is expected, but in the case where the model is externally provided and unknown (such as the VTuber project) it would be even more cumbersome, if possible at all.

Is there a reason why this should be core and not an add-on in the asset library?

Cannot be achieved by scripting alone (either GDScript, GDNative or even module) as it requires changes in VisualServer and rasterizer.

[clayjohn]

I think this is a great idea! I can definitely understand the pain of trying to coordinate next_pass style materials across many surfaces.

I have two comments/questions:

1. What do you think of a name that somehow conveys that this is essentially a next_pass material for the entire mesh? I like material overlay (as it is a good descriptive name), but it is essentially the same feature as next_pass only applied to the whole mesh. I don't love the idea of having two similar features with very different names. Perhaps we could brainstorm something more related to next_pass, like instance_next pass or something similar.
2. I think if we add this to 3.x, we need to add it to 4.0 at the same time.

[fbcosentino]

Not really sure about point 1. A next pass material is added as next pass to another material, while the overlay does not belong to any material in particular. It stands by itself, just not discarding what was rendered before. In other words, it is not part of the cascading "while" belonging from a previous material, it is rendered as a normal material, it just happens after the normal material (and it also has its own next_pass while loop). It's more like "subsequent pass".

I fully understand the intention to be convey an analogy to next pass materials since it works similarly, but I'm just afraid that would end up less semantic, even confusing. Someone not used to using next_pass materials and looking at that property and trying to guess what it does, "instance_next_pass" doesn't really convey what it does. We already have material_override which is very semantic, "material_overlay" was chosen to be a pair to it, following the same logic.

Fully agree adding it to 4.0. Not necessarily at the same time as I don't understand the implications in Vulkan, but will gladly do it as soon as I understand how to do it.

[reduz]

Would it be enough with a property in GeometryInstance material override to make it multipass? (could be a mode like replace/overlay)

[fbcosentino]

It would work, but is there any advantage?

The code locations to be changed are the same and it also adds a property to GeometryInstance (so there is no gain in simplicity, in either development or user side), and reduces the features, as you can no longer use an overlay on top of an override independently. An effect would need to use script code to toggle between the the overlay mode directly versus the override mode with the overlay as next pass of the override material if the target has a material in override mode, and AnimationPlayer tracks have to be updated accordingly.

[sairam4123]

Related to #2595

[Calinou]

This was implemented in 4.0 by godotengine/godot#53069 and in 3.5 by godotengine/godot#50823.