Add support for stencil buffer

[UsernameBillion]

Describe the project you are working on

I'm working on a game that visually I want to resemble old 2000's games. So I wanted to make shadow volumes with a shader and some gdscript, but I can't.

Describe the problem or limitation you are having in your project

Without a stencil buffer, I can't get it to work. I've got very close by using depth and/or alpha + multiple passes but there is always some kind of flaw that makes it unusable for the project.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With a stencil buffer (...or a way to alter a previous pass), I'd be able to make it work.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I'm not sure how a stencil buffer works if I'm honest, I just know thats what I'd ideally need.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, I can't find a single way to do this with the tools godot provides.

Is there a reason why this should be core and not an add-on in the asset library?

I don't know if something like this COULD be added as an addon?
And I think simply having a stencil buffer seems like a reasonable thing to want as a core feature.

I know the plan is to add custom buffers or something in 4.0 but I'm making this as a request for at least a stencil buffer to be ported into 3.x. I'm one of the people who doesn't want to move to 4.0, this would be my one sole reason to do that and I don't think its worth the trouble.

[Calinou]

This was already discussed in a proposal a while ago, and also in an issue: #1930, godotengine/godot#23721

The consensus is that built-in stencil shadows will not be added, but a stencil buffer may be exposed in a future 4.x release (not 4.0 due to time constraints). This stencil buffer can then be used in custom shaders to create stencil shadows and other effects.

but I'm making this as a request for at least a stencil buffer to be ported into 3.x.

The 3.x branch won't get large new rendering features due to the work required to implement them. This is especially the case for features that can only be made to work in the GLES3 backend.

In the meantime, consider using blob shadows using a Sprite3D + RayCast setup (or in 4.0, a Decal node) 🙂

[UsernameBillion]

This was already discussed in a proposal a while ago, and also in an issue: #1930, godotengine/godot#23721

The consensus is that built-in stencil shadows will not be added, but a stencil buffer may be exposed in a future 4.x release (not 4.0 due to time constraints). This stencil buffer can then be used in custom shaders to create stencil shadows and other effects.

but I'm making this as a request for at least a stencil buffer to be ported into 3.x.

The 3.x branch won't get large new rendering features due to the work required to implement them. This is especially the case for features that can only be made to work in the GLES3 backend.

In the meantime, consider using blob shadows using a Sprite3D + RayCast setup (or in 4.0, a Decal node) 🙂

I'm not at all saying you're wrong since I know next to nothing here, but why would adding a feature already within gles be difficult? Doesn't godot simply convert a godot shader into a gles shader internally, wouldn't that be about as complicated as adding some string-altering code?
I'm also using GLES2, not 3.

[Calinou]

but why would adding a feature already within gles be difficult?
Doesn't godot simply convert a godot shader into a gles shader internally, wouldn't that be about as complicated as adding some string-altering code?

It's not that adding the feature is difficult. We consider the current GLES renderers to be in maintenance mode, and we'd like to avoid introducing regressions in them. We also need to be 100% sure that any features added to them can be replicated in Vulkan, so that projects can be ported from Godot 3.x to Godot 4.x without too much effort.

Implementing a stencil buffer in Godot is more than just adding some lines of shader code. A convenient API needs to be designed, in line with the Best practices for engine contributors #6. We also need to ensure that there's no performance cost when you don't make use of the feature in a project (in a way similar to godotengine/godot#38926).

I'm also using GLES2, not 3.

I presume GLES2 doesn't mandate stencil support (or only in limited form), which makes things even more difficult. We don't want to add desktop-only features to the GLES rendering engines, as they are intended to work on mobile/web platforms as well.

[LunaTheFoxgirl]

I plan to eventually have ports of Inochi2D to off the shelf engines like Godot, so bumping this since Inochi2D extensively uses the stencil buffer to handle constraining rendering of parts to other parts (eg. forcing the iris to stay within the sclera (eye whites) for eye rigging, forcing shadows to only overlap specific parts, etc.), shader based alternatives to stencil buffers would be considerably slower since they're used so often.

[elvisish]

I think a stencil buffer would be needed to allow shaders to place materials on top of faces behind them. For instance, billboarding 3D sprites explosions clip partially into walls if the player isn't looking directly at the wall it's been instanced at, and disabling depth test just makes the sprite appear on top of everything, including geometry in front of it. As far as I know, a stencil buffer is needed to write a shader that can perform a depth test and always keep the billboard sprite on top of faces behind it.

[Calinou]

For instance, billboarding 3D sprites explosions clip partially into walls

You can enable proximity fade in SpatialMaterial to make it blend softly with solid surfaces.

[elvisish]

For instance, billboarding 3D sprites explosions clip partially into walls

You can enable proximity fade in SpatialMaterial to make it blend softly with solid surfaces.

If it's an animated 3D sprite, there isn't any way to use spatial material, is there? Since it would have animation frames, I'm not sure how that could work. I tried adding a Material Override with an AnimatedTexture, but it didn't seem to work.

[alfredbaudisch]

A proposal in another repository regarding Outlines and Silhouettes, which probably requires stencil buffers alfredbaudisch/GodotShaderCollection#3

[Zireael07]

Yes, most outlines require stencil buffers.

[kayomn]

Another use-case for stencil buffers involves covering the holes in meshes created by cross-sectioning the geometry in shaders.

If the stencil buffer were exposed, I would be able to project a textures like albedo and normal map atop where the backface is visible to the camera to create effects like a liquid surface.

This is theoretically possible without a stencil buffer, but far more difficult.

[apples]

I managed to implement this fairly straightforward, at least for Forward+.

See my related comment here: alfredbaudisch/GodotShaderCollection#3 (comment)

Godot already has a working stencil buffer of course, and all it took was exposing the relevant PipelineDepthStencilState fields to BaseMaterial3D, and adding render modes for them.

I suppose the question is whether or not that's a good enough solution. It's easy to imagine scenarios where per-material settings aren't very convenient, such as portals.

Additionally, it's not super beginner-friendly. I suppose the stencil buffer never is, but it'd be nice to have a simple StencilMaterial or something to do basic outlines.

[clayjohn]

We have been discussing this in a bit of detail on Rocketchat. So I would like to summarize some of the discussion and provide my input on this.

Some general comments

1. Stencils are something that we need to support eventually. They are super useful and we have no reason not to include them.
2. The reason we haven't included stencils so far is it is very difficult to decide how to expose them to users. Stencils are very complicated, but most users don't need the full breadth of functionality.
3. Most users appear to need stencils to do one of three effects 1) outlines, 2) stencil shadows, 3) x-ray/silhouette effect. We could implement these effects without exposing stencils (this was reduz' recommendation years ago)
4. Eventually we want to support an OverrideShader type that allows users to fully write custom GLSL shaders (instead of using the Godot SL that gets inserted into our ubershader, you would start from the ubershader and tweak it however you see fit). This OverrideShader would allow users to control all the details of the pipeline and do much more advanced effects.

My thoughts:

1. We should expose a minimal set of stencil operations that allow users to do most effects without exposing every single stencil operation. For example, instead of exposing blend function and blend equation, we only expose 5 "Blend Modes" which internally are combinations of blend functions and blend equation (for reference, there are 19 possible blend functions and 5 different blend equations, blend functions can be specified for source color, source alpha, dest color, and dest alpha. Accordingly there are 19 * 19 * 19 * 19 * 5 = 651605 possible combinations and we expose only 5 of them to users). A few users have requested alternative blend modes, but most users are more than happy with what is exposed.
2. IMO the important thing here is to make it so that users can easily use stencil to do outlines, stencil shadows, and x-ray effects. Since these 3 effects will make up 99% of use-cases, they should be prioritized. Then we should leave the door open for more advanced use cases by allowing users to write more customized shaders.
3. We need to figure out how stencils should interact with the depth prepass. i.e., can we get away with only writing to stencil during the depth prepass and only reading during the opaque pass (transparent of course has to read and write in both).
4. We also need to discuss whether the current render_priority field is enough to solve ordering issues. Stencils often require having a specific order of renderer, but Godot sorts both the opaque pass (by state) and the transparent pass (by depth).

[LunaTheFoxgirl]

This would be insufficient for Inochi2D, but I've found a workaround based on writing my own renderer "from scratch" using RenderingDevice exposed in Godot 4. Does mean some platforms won't be supported but there's nothing I can do about that I think.

[IceReaper]

I'm also interested in using the stencil buffer to pixelize single objects (make models look like pixel perfect 2d 8bit 32x32 sprites). Having this buildin or allowing this setup from the shader would be highly appreciated :)

[InitialCon]

I think that while those 3 effects make up the majority of use cases the engine should still support custom usage of the stencil buffer as it makes several effects much easier to achieve. Speaking from my own perspective I'm currently investigating using the stencil buffer for an underwater effect so even just exposing bitwise operations on the stencil buffer would be incredibly useful. An example of this would be exposing the 8 bits of the stencil buffer we currently allocate as a texture and allowing materials to write to a particular bit of the stencil buffer when drawn. This would make most effects simple to implement given enough effort while allowing the most common effects to still be included by default by reserving the first few bits if the community decides that they want them.

[QbieShay]

A little extra on the use cases, I don't know how i forgot them before.
There's this technique on VFX that needs stencil which is when you have a portal of some form and you want stuff to show through only when the objects are visible through the portal.

examples of this are

1. Impossible cubes with multiple things inside: https://www.youtube.com/watch?v=EzM8LGzMjmc
2. holographic cards: https://simonschreibt.de/gat/the-unreal-stencil-dragon/
3. portals in general (can't find examples right now).

So it would be quite important to also be able to have geometry intersect with stencil surfaces (like n2)

[QbieShay]

@LunaTheFoxgirl could you expand on your usecase?

[LunaTheFoxgirl]

@LunaTheFoxgirl could you expand on your usecase?

My usecase is in a 2D character rendering system that I develop. Stencil buffer is needed in a fixed function manner where I can before rendering a "part" set up masks by cutting in to and out of the stencil buffer. This allows the end user of my software to for example keep the iris of their character only rendering when it overlaps the sclera. Or create otherwise complex rendering masks.

[QbieShay]

I see. I don't think it's possible to use the raw stencil function from the gpu for 2D object (as far as i know stencil is set only for the full mesh, and you can't use transparent textures with it?) but should be solved instead by the clipping of canvas items. That should already be available in Godot4 without using stencil buffer

[apples]

3. Most users appear to need stencils to do one of three effects 1) outlines, 2) stencil shadows, 3) x-ray/silhouette effect. We could implement these effects without exposing stencils (this was reduz' recommendation years ago)

I feel like it's easy enough to implement these with basic stencil ops and depth functions. That said, an easy to use XrayOutline node or something would be great to have, though that could be implemented by an addon.

3. We need to figure out how stencils should interact with the depth prepass.

For any stencil effects which do also write depth, it would be incorrect to draw depth for pixels which would later be stenciled out and not drawn. However, xray/outline effects do not usually draw depth, so this won't affect them.

4. We also need to discuss whether the current render_priority field is enough to solve ordering issues. Stencils often require having a specific order of renderer, but Godot sorts both the opaque pass (by state) and the transparent pass (by depth).

First, as a user, I expect render_priority to always be the first thing that rendering is ordered by. No matter what else happens.

I do think that an additional sorting solution could be found, but I personally feel it's unnecessary.

Maybe a good solution would be to have a system of named layers, arranged in the project settings, which automatically set the render_priority for materials that use them.

[LunaTheFoxgirl]

I see. I don't think it's possible to use the raw stencil function from the gpu for 2D object (as far as i know stencil is set only for the full mesh, and you can't use transparent textures with it?) but should be solved instead by the clipping of canvas items. That should already be available in Godot4 without using stencil buffer

Canvas item clipping is not suitable for the needs of Inochi2D, as I need to be able to cut in to and out of the mask by re-rendering items that was previously rendered in to the stencil buffer; but as I said RenderingDevice does expose the needed features.

My plan is to just add a Inochi2D puppet node type which internally renders to 3 textures which then can be used in conjunction with a mesh and material in godot.

[QbieShay]

Thank you for your explanation @LunaTheFoxgirl !

[jordo]

anecdotally, +1 vote for exposing stencil buffer for full usage, instead of just being limited to: outlines, stencil shadows, x-ray/silhouette effects.

[QbieShay]

Since it'll be a bit before 4.x is open for features (4.1 release) can we put together a list of stencil operations we want andconsider minimal enough feature set?

I think in order to make stencil more easy to use we could have preset configurations and leave a "custom" option that unfolds the whole option set?

[thompsop1sou]

These have already been mentioned, but i'm hoping to be able to use stencil shadows for a project i'm working on (and silhouettes would also be nice), so +1 vote for those features.

[QbieShay]

Spent some time looking at how Unreal and Unity do it so I can finally propose a workflow for this.

This is a proposed 3D workflow. I have no idea how to use it with 2D. I wouldn't support 2D for now.

Stencil is composed of three components: write, test and read.

Write

Stencil write as far as i understand is severly limited by the mesh: either you write the whole mesh to stencil, or you don't.
I saw that Unreal doesn't do stencil write in the shader, but rather in the mesh component and i think this is a very good approach that we should replicate.

So, on GeometryInstance Material add a "Stencil" section that looks like this

EDIT: updated with increase/decrease op

A bit of info in that mockup:

• Write number and write mask are mutually exclusive fields that will be displayed based on wheter write mode is integer or bitmask
• All will be hidden unless "Enable" is set to true
• write condition is important for Xray shaders.

Test

Test configuration should be on the Shader Material resource. We can add it also to spatialmaterial later if people request it, but this is supposed to be an advanced feature so i expect people to use it with custom shaders.

Once #7117 is implemented it will make more sense to see stencil as a property of the shader rather than a render_mode. I don't think we should support it in a render mode unless it's strictly necessary. It adds overhead to the shader compiler and i don't think it's necessary since this is a pipeline state (?) and not a shader program property (correct me if i'm wrong on this, i'm severly out of my turf here)

As above, compare op and compare mask are mutually exclusive fields.

Read

Add a shader built-in called STENCIL (integer)
Add a shader built-in called STENCIL_MASK which is an array of zeros and ones that reflect the bits of STENCIL

For visual shader I don't know yet what to expose, but i imagine the tricky part is bitmask comparison? To be defined how this is used, for the time being it can be an expression block.

Considerations

This is mostly inspired by Unreal's stencil because honestly I couldn't understand Unity's one, but I tried to add a little extra to what Unreal offered and to adapt it to Godot's structure. I don't know if there's any effect that is not doable with this setup, please let me know.

[MJacred]

@QbieShay: I'm also out of my turf here, but I'd really like to have stencil buffers for fake lighting like in wind waker. So to put in my two-penny worth…

MinionsArt explained it here for Unity in greater detail than Simon's paper which the video is based on (might help you understanding Unity's way).

The way I understand your proposal,

• in write, you can only do the replace operation (Unity)
  • for use cases of other operations, such as IncrWrap, see video before 4:30
• you cannot check for a specific value using comparison and then write a specific different value back (around 6:45-7:15 in the video, we check for value 1 and need to overwrite with zero). You only have your write number / write mask value
  • so from the 3 spheres needed, we'd fail at sphere 2. Sphere 3 would fail for the same reason

What I wonder about is at 5:14-5:45 you require an interaction with z testing. And afaik Godot can only do sth. similar using next_pass (#496 (comment)). No idea if that step would work

And no idea how to translate ColorMask to Godot's way

[QbieShay]

I wonder if these lights have to be done with stencil. I think it should be possible to achieve a similar effect?

This seems very specific but I can adapt the stencil test mockup part

After looking, this should be possible to do by combining stencil test aabd depth. You'd do stencil test on bigger than, then stencil write with value.

Adding increment and decrement to stencil write as a mode should work?

[MJacred]

I wonder if these lights have to be done with stencil. I think it should be possible to achieve a similar effect?

Achieve sth. similar using normal lights? Maybe… Speaking for our stylistic context, I don't know the limits of Godot's light projection to enforce a certain shape (octagon or sth.) and I'd need to test if I can output the pure diffuse texture rgb values unaffected of shadows if no light is hitting the object (cannot use unlit/unshaded if we use normal lights)…

Adding increment and decrement to stencil write as a mode should work?

Do you mean: "I'm checking if the value is 0, and if it is, increase/decrease by 1" ?

I'm no shader wizard, sadly, but my programmer heart tells me Unity's Stencil operation values are a good way (also in combination with a different write operation based on whether stencil passed, failed or depth test failed).
Same with the other comparison values: NotEqual and Always, though I don't understand the use of Never.

If I understand your proposal correctly, the write operation is only done if the depth condition AND stencil test are both true?

I'll take a look at Unreal's approach tomorrow; they should have sth. similar…

[QbieShay]

yes, that's the idea.

Unreal's one is much simpler than Unity, exposes a lot less knobs. I'd prefer to sit somewhere in the middle with Godot, because Unity's one is really too hard to understand. unreal's one in comparison is veeeery simple but i fear also very limited.

[QbieShay]

@clayjohn pointed out in chat that having it on the mesh itself is a bad idea since it would prevent using multiple draw passes with different stencil operations, de-facto invalidating the desired workflow of easy to do outlines.

Stencil write should probably be moved on Material as well.

[MJacred]

So, I took a short look at Unreal's approach and noticed that it goes wholly via post-processing. It's completely different to Unity's approach. So they can offload a lot of logic to the post-processing shader. Not sure what kinds of repercussions that has.
Based on the conversation so far, Godot goes into the same direction as Unity (putting it into the geometry material/shader).

From what I read and seen so far, the stencil buffer is for creative use of masking effects. Simplifying it more than what Unity does feels wrong to me… If most users just want the "x-ray" and "outline", they can use the G-Buffer (geometry buffer) according to this comment. If they still need the stencil buffer because the g-buffer is not enough, then for just those 2 use cases, writing tutorials sounds the best way to go.

That's it from me for now, I don't want to ping everybody any more than necessary.

[QbieShay]

No,the g buffers are not sufficient to do these effects. Also the PR you linked is not merged.

As for the lights of wind waker, it's a really specific usecase. I have a feeling it should be possible to achieve it in some other way with a light shader 🤔 but regardless i believe that it should be achievable with what I proposed. replace is already possible with the options we're exposing. I think that stencil buffer should allow to read and write in the same moment. So you'd enable test, test against previous stencil, and enable write. that should achieve everything, i think? I had a hard time following the video x)

[MJacred]

I'd have to rewatch it again (or better, reading it :https://www.patreon.com/posts/wind-waker-style-78831006) to see if the required operations are covered. It really is not that easy to wrap your head around. But complex uses never are x) and then there's the question about the ztest and colormask... I'll try out a stencil PR to see if it's achievable once we have one. And yes, read and write at the same moment is must

No,the g buffers are not sufficient to do these effects. Also the PR you linked is not merged

I referenced someone who should know better than I, so I cannot comment on g-buffer's usefulness... And the PR is readier than stencil support: Only vulkan version is missing Am 21. Juni 2023 21:44:49 UTC schrieb QbieShay ***@***.***>:

…

No,the g buffers are not sufficient to do these effects. Also the PR you linked is not merged. As for the lights of wind waker, it's a really specific usecase. I have a feeling it should be possible to achieve it in some other way with a light shader 🤔 but regardless i believe that it should be achievable with what I proposed. `replace` is already possible with the options we're exposing. I think that stencil buffer should allow to read and write in the same moment. So you'd enable test, test against previous stencil, and enable write. that should achieve everything, i think? I had a hard time following the video x) -- Reply to this email directly or view it on GitHub: #3373 (comment) You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[jordo]

This is a proposed 3D workflow. I have no idea how to use it with 2D. I wouldn't support 2D for now.

Awe too bad, that's super disappointing to hear :(

[QbieShay]

@jordo I understand the disappointment, but please understand this work is already complicated :D when I said "I wouldn't support 2D for now" doesn't mean "we'll never support 2D" but rather "I'm already overwhelmed trying to make sense of this in 3D that I don't have the bandwidth to deal with the 2D side"

There's different considerations to be done there, namely there is no depth prepass in 2D so there isn't a proper place to write to stencil yet, which is just yet more stuff to think about. I want to approach this in a step by step manner. I hope that clears why i said "no 2D for now" ^^

[jordo]

@jordo I understand the disappointment, but please understand this work is already complicated :D when I said "I wouldn't support 2D for now" doesn't mean "we'll never support 2D" but rather "I'm already overwhelmed trying to make sense of this in 3D that I don't have the bandwidth to deal with the 2D side"

There's different considerations to be done there, namely there is no depth prepass in 2D so there isn't a proper place to write to stencil yet, which is just yet more stuff to think about. I want to approach this in a step by step manner. I hope that clears why i said "no 2D for now" ^^

Ok I see. I guess I kind of thought the stencil interface/api would at least be exposed by the rendering server/layer, (or whatever the equivalent is in 4.0 is now renderingdevice???), so that we can use it fully there. Which I thought may likely be simpler in 2D than in 3D.

[QbieShay]

I am not sure how'd it work for 2D, but i personally specialize in 3D so I can't be of much help there.

[QbieShay]

Closing in favor of #7174