Implement a Rendering Compositor

[reduz]

Describe the project you are working on

Godot 3D Rendering

Describe the problem or limitation you are having in your project

There are a good number of rendering techniques that can't be used in Godot due to the relatively monolithic renderer design.

As examples:

• Fine grained stencil control.
• Portals.
• Planar reflections.
• Geometry decals, since you can write and read back custom buffers.
• Terrain blending to geometry.
• Geometry blending to sky.
• Custom post processing.
• And a few others

Godot renderer takes objects, renders them and that's it. To be able to implement most of these techniques, multiple, customizable opaque passes are required.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to implement a compositor. Basically you create a RendererCompositor resource and set it into either WorldEnvionment or Camera nodes.

The compositor lets you:

• Allocate custom buffers for custom rendering
• Divide opaque rendering into several passes.
• Assign materials to each pass.
• Read and write from/to custom buffers in materials.
• Do clears and copies to backbuffers in each pass.
• Do stencil operations in each pass.
• This may not be enough for all cases, so for the more exquisite render engineers, give low level access to everything going on and allow inserting custom rendering code at any stage of rendering and between passes.
• Additionally, also add a rendering logic script to control dispatch (call render_camera internally with custom parameters) which allows to do effects such as portals or planar reflections.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Here is an API draft at RenderingServer level (for users, there will be an easier to use resource exposing this in a more friendly way).

// Following are methods of RenderingServer

RID compositor_create();

// This lets you add custom buffers to rendering, accessible in materials for reading and writing

enum CompositorCustomBufferFormat {
    COMPOSITOR_CUSTOM_BUFFER_DISABLED,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_R8,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RG8,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RGBA8,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_R16F,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RG16F,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RGBA16F,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_R32F,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RG32F,
    COMPOSITOR_CUSTOM_BUFFER_FORMAT_RGBA32F,
};

enum {
    COMPOSITOR_MAX_BUFFERS = 4
};

void compositor_set_custom_buffer_format(RID p_compositor, int p_buffer_index, OpaqueMultipassCustomBufferFormat p_format); // max 4 buffers.

// This lets you add opaque passes to the compositor.
// pass 0 always exists and can't be removed.
// Materials without a pass defined go to pass 0

void compositor_add_opaque_pass(RID p_compositor, int p_index); // if exits it does nothing.
void compositor_remove_opaque_pass(RID p_compositor,int p_index);

enum CompositorOpaquePassActionFlags {
    COMPOSITOR_OPAQUE_PASS_ACTION_COPY_DEPTH_TO_BACKBUFFER = 1,
    COMPOSITOR_OPAQUE_PASS_ACTION_COPY_SCREEN_TO_BACKBUFFER = 2,
    COMPOSITOR_OPAQUE_PASS_ACTION_COPY_SCREEN_MIPMAPS_TO_BACKBUFFER = 4,
    // Pass 0 has these all on by default
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_STENCIL = 8,
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_DEPTH = 32, 
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_CUSTOM_BUFFER0 = 64, 
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_CUSTOM_BUFFER1 = 128,
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_CUSTOM_BUFFER2 = 256,
    COMPOSITOR_OPAQUE_PASS_ACTION_CLEAR_CUSTOM_BUFFER3 = 512,
    // No clear screen since actual clearing happens during the bg_clear pass (see compositor_set_background_clear_pass_index)
};    

enum CompositorCustomBufferMask {
    COMPOSITOR_CUSTOM_CUSTOM_BUFFER0 = 1,
    COMPOSITOR_CUSTOM_CUSTOM_BUFFER1 = 2,
    COMPOSITOR_CUSTOM_CUSTOM_BUFFER2 = 4,
    COMPOSITOR_CUSTOM_CUSTOM_BUFFER3 = 8,
}

void compositor_set_opaque_pass_action_flags(RID p_compositor,int p_pass,BitField<CompositorCustomBufferMask> p_flags);
void compositor_set_opaque_pass_stencil_clear_value(RID p_compositor,int p_pass,int p_value);
void compositor_set_opaque_pass_depth_clear_value(RID p_compositor,int p_pass,int p_value);
void compositor_set_opaque_pass_custom_buffer_usage(RID p_compositor,int p_pass,BitField<OpaqueMultipassCustomBufferMask> p_usage);
void compositor_set_opaque_pass_custom_buffer_clear_color(RID p_compositor,int p_pass,int p_buffer,Color p_color);
void compositor_set_opaque_pass_render_depth_prepass(RID p_compositor,int p_pass,bool p_enable); // For forward renderers with depth prepass, do the prepass for objects meant to be drawn here _and_ in subsequent passes that have this flag disabled. Enabled by default.

void compositor_set_background_clear_pass_index(RID p_multipass,int p_index); // If the index is equal to existing pass, sky or clearing should render at that pass, else at the end. Default index is 100.


// This lets you add custom rendering logic to compositor. Lets you trigger multiple camera renders (needed for portals or planar reflections)
// The supplied callback will need to call a function `render_camera` with relevant arguments to trigger a renderer.
// It can be called multiple times.
void compositor_set_custom_camera_logic(RID p_compositor, const Callable & p_custom_logic_callback); // max 4 buffers.

// This lets you add custom rendering effects to the compositor (see effects proposal)
void compositor_add_effect(RID p_compositor_effect); 
void compositor_remove_effect(RID p_compositor_effect); 

/.../

void environment_set_compositor(RID p_environment, RID p_compositor);

On the material side, the following new features exist:

shader_type spatial;

compositor_opaque_pass 2; // default is zero

//uniform sampler2D buffer_texture0 : hint_custom_buffer0_texture, repeat_disable, filter_nearest;
//uniform sampler2D buffer_texture1 : hint_custom_buffer1_texture, repeat_disable, filter_nearest;
uniform sampler2D buffer_texture2 : hint_custom_buffer2_texture, repeat_disable, filter_nearest;
uniform sampler2D buffer_texture3 : hint_custom_buffer3_texture, repeat_disable, filter_nearest;

void fragment() {

  CUSTOM_BUFFER0 = vec4(1,0,1);
  CUSTOM_BUFFER1 = vec4(1,1,1);
  //CUSTOM_BUFFER2 = vec4(0,0,1);
  //CUSTOM_BUFFER3 = vec4(1,0,1);
  
}

If compositor_opaque_pass is defined, the material will always be opaque, even if it reads from screen and depth.
An excellent property of defining this in the material is that this workflow remains entirely compatible with GPU driven rendering (where materials are dispatched instead of geometry).

Some bits:

• Materials cant write to a custom buffer and read from it at the same time (shader compilation error).
• On render time, if the material does not match the pass format (buffers for reading or writing), the pass is dropped.
• Layout changes to sample-read (from render target to sampling) on subsequent passes can be done only if any material actualy reads them.
• Reading from custom buffers should work fine during the alpha pass. No limitation.

High level exposure

As mentioned before, resource RenderingCompositor can be created that lets you edit the values above (passes, indices, custom buffer formats, etc).
It is assignable to WorldEnvironment or Camera. Should it be a separate resource to Environment? I think so.

How does it work under the hood?

The general ideal is that the opaque pass as it happens now needs to be left as is. We have several optimizations where as an example, with TAA we draw objects that moved first, then objects that did not move. On deferred, it will be several more optimizations like this.

So passes needs to happen at even coarser level like:

[pass: <actions> <all_rendering>] [pass: <actions> <all_rendering>] [pass: <actions> <all_rendering>]

In order to not change general rendering.

Custom code can be run between the passes (or at any other time) using a CompositorEffect (more on this below).

Compositor Effects

The idea is that this #80214 is adapted to be a CompositorEffect class that can be added inside RenderingCompositor instead of Environment.

Examples

What are examples that can be implemented how would they be used?

• Stencil Effects
  • Cutout
  • Cracks
  • Object Masking
  • Bounding
• Portals (using two custom render logic calls to render_camera AND custom buffer usage)
• Planar reflections (using two custom render logic calls to render_camera)
• Geometry decals (using a pass between depth prepass and color pass that writes the decal information to a custom buffer).
• Object outlines (writing object and object depth to custom buffers, then using a compositor effect to draw the outline)
• Terrain blending to geometry (using an additional pass after a depth copy, then using blending or jittering depending on render backend)
• Geometry blending to sky (using an additional passa after sky is rendering and a screen copy pass)
• Custom post processing (using compositor effects at any point in the chain even between passes)

Platform support

Should work in all renderers, including compatibility. CompositorEffect, however, is platform dependant. May be a possibility to create a new shader type, "compositor" to do some of the most common operations users ask for without having to wirte low level rendering code.

FAQ

Q: Why not a compositor graph?
A: We gave this idea a spin many times and really tried to make the case for a compositor graph. But, to be honest, in far most cases you just want to plug something and expect it to work as intended. A graph for an engine like Godot would make it harder for non-rendering folk to use this because you would not understand where to plug compositor effects.

Q: Why opaque passes and not transparent?
A: There is not much of a point in doing this with transparent passes. Transparency is sorted from back to front and renderer within a single render pass. Additional, there is not much of a point of transparency writing to custom buffers because blending is active (well sure you can blend but we enter weird corner case territory). Reading from the buffers written by opaque passes from the alpha passes of course will be permitted.
Q: Can you give more detail on how custom render logic will be implemented?
A: This needs to be defined better.

Q: Can I assign a material to more than one pass?
A: No, but you can create multipass masterials (a proposal opened to do this more easily can be found here).

Q: Why is the pass assignment material based?
A: Besides being easier to implement (and good for usability) this is fully compatible with GPU driven rendering (that we want to implement in the future)

Q: Why can't I clear the color buffer in an opaque pass?
A: The clearing happens during the clear operation, which draws the sky or a flat color on the pixels that were not drawn yet. This is done for performance. Additionally, when using deferred rendering, there is no color to be found in the opaque pass.

Q: Does this not proposal miss a lot of flexibility?
A: This proposal is designed to fulfill the goal of allowing to resolve all the use cases mentioned above. It provides an interface to very easily solve common opaque pass effects, while leaving the door open to do anything you want using composite effects on the low level.

Q: Who is this proposal aimed for?
A: Primarily, game developers with some basic understanding of rendering, technical artists and VFX artists. It means to be simple to use and understand. For those who are more into the bare metal, the composite effect interface provides more flexibility.

If this enhancement will not be used often, can it be worked around with a few lines of script?

N/A

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[forestrf]

As someone coming from Unity, this sounds similar to the Scriptable Rendering Pipeline they have. I think my best suggestion would be to have all the passes be reorderable and configurable, with the ability to add or remove existing and custom passes. This would lead to the ability for the developer to break features like TAA as pointer out in the proposal, but in exchange would also allow disabling those features for those who don't need or want them to achieve their vision. Those features that require certain passes in a certain order should check for the existence of such passes and not run if they aren't found. Given custom passes could perform a task compatible to the passes TAA and other features may need, passes may include flags to signal what they do or to show that they are equivalent to an existing built-in pass, so that TAA and other features that require such passes know they can work.

I'm sorry as I don't know if something similar to this exists in Godot, but Unity's CommandBuffer API is an interesting thing I like because of how generic it is. Unity's implementation is far from perfect but seems like a good API to write this custom passes (https://docs.unity3d.com/ScriptReference/Rendering.CommandBuffer.html). The most important features I find from such an API (some of this features aren't in Unity, this is a wishlist) are:

• The ability to provide a list of renderers (or get the list of renderers) to work with.
• Cull/filter the renderer list by layers, material and shader properties, camera's frustum and portal visibility.
• Sort the list of renderers front to back, back to front, by shader, by a material property (in Unity this can only be one called RenderQueue) and more.
• Specify a material property override for all of them.
• Finally render them with their materials or specifying a custom material and/or shader.
  I think this may be difficult when taking into account lights and shadows. I think nodes should be used more instead of doing it all by code as it may simplify the logic, so things like adding renderers or lights from code may not be the best idea even though that's what I just said before.

I'm afraid that if the API given to customize the pipeline is too high level it will limit what it can be done with it.

[kebabskal]

Would this also make it achievable to for example render 3d scene gizmos after post processing?
Using the RenderingEffect PR #80214 I have a problem where my custom effects (such as pixelization and color quantization) are applied to the scene gizmos, making it impractical to edit with the effects on.

From what I understand, this is not easily doable with the current pipeline.

[reduz]

@kebabskal yeah, you can basically dispatch two camera render passes, the second oned use to draw gizmos. How this API will work is uncertain yet, though.

[reduz]

@forestrf The idea here is to make something simpler than what Unity has (as in, you wont be able to make your custom TAA or custom GI most likely), but at the same time if you want to add custom render logic, you can hook your own code directly with the 3D API. There you have full freedom.

[strich]

@reduz given your answers so far I think its out of scope for your proposal but just for clarity - It has been quite common for me to write a custom render pass in Unity to support some unique render feature in our games. For example object highlighting/outlining, or bespoke hole cutting with proxy meshes.
Due to the way Unity has designed their post processing stack it was relatively easy and powerful to do, as well as easily get plugins for other common things (bloom, tonemapping, custom 2D widget drawers, etc):

An additional side effect of this design as that it is easy to create unique stacks for different platforms or performance targets. For example our console build has almost all those effects disabled and out of the loop.

Like I said it sounds like this is not the kind of design you're currently going for - I just wanted some real-time usage examples if that helps you figure out where these features should go.

[csubagio]

Hmm, this is interesting. There's some light jargon here that I think is a general mess in the games industry, so I'd caution folks to be careful with assumptions based on the names of things alone. Having said that, my assumption here is that you're aiming this solution at specifically: rendering features that need to render some part of, or all of, the same scene more than once, capturing the intermediate results in texture buffers that will be used in subsequent steps?

My primary worry with your proposal is that it might not be enough, but only because as a render engineer, my ideal would likely be different, and I'm still processing how to fit that into your framework. To try and explain in brief, I abstractly think of this as being a kind of "render pass" which I define as a unit of rendering work that takes a scene and produces a set of buffers (texture or otherwise). In the course of that work, a pass filters which draw calls in the scene are used, in what order, and what configurations need to apply to the materials. The material system cooperates with passes to efficiently expose what needs to be configured, e.g. making dependent data like lights available, changing aspects of shared vertex processing like projection matrices, and channeling outputs from varied shaders into semantically equivalent things, e.g. color or normals. In my head, rendering a shadow buffer, a z-prepass, a depth buffer occlusion test, an opaque deferred pass, a forward pass, a hard split between transparencies inside and outside of a cockpit, an intermediate blurring downsample step, and a applying a combined tonemap/bloom, all fit into the same pass mechanism.

Godot already accomplishes a number of those features, e.g. shadow mapping and separate opaque and transparent passes. What raises a red flag for me with this proposal, is that it doesn't unify all of these concerns into a single concept, because in the past, engines I've worked on have ended up in a sticky situation when they find areas where overlap between these features cause redundant or even competing mechanisms to exist, without a clear way to resolve them. Sorting, is usually one, but then material definitions and permutations usually get involved. Then there's flagging of objects/materials/groups for inclusion/exclusion in various sets of things that start becoming confusing to name in the UX. It leads to the kind of "Oh, that's a material pass and not a depth flag? Wait, why is this a material layer flag instead of a pass? Oh, because it's transparent? Ok, I guess that makes sense." discussion where you're really just exposing the engine layout to the user, rather than the feature name.

Anyway, the tldr: I would suggest some time considering whether there's a refactor of the existing "pass like" features of Godot, which would result in a new composable "pass" primitive, that could then also accommodate the new "pass like" features you describe, in such a way that they touch the scene, node, material, buffers, and sorting systems in uniform ways.

[darksylinc]

Oooff where do I start.

-1. 4 textures is nowhere near enough for some effects. Period. If you keep it to CUSTOM_BUFFER0 to CUSTOM_BUFFER3, it will be a cute FX system that few users will use. It is too restrictive.

For example of how flexible it should be, it should be possible to implement HDR (HDR rendering + luminance reduction + blur/bloom + final tonemapping that merges everything).

Additionally, hashed strings are a much more user friendly way here (e.g. "Luminance Texture")

-2. There's no way to add external textures. For example, if the user wants to bind a noise texture created by hand to apply on the main target; or an UBO with baked parameters.

-3. I don't like that textures are called "buffers". This is an ancient naming convention from OpenGL in the 90's where the window would be called the "frame buffer". However nowadays "buffer" means UBO, SSBO, TBO; while texture encompases sampling, render targets, render windows and UAVs (image textures in Vulkan lingo).

-4. CompositorCustomBufferFormat has no gamma support.

There is not much of a point in doing this with transparent passes.

-5. Extreme disagree. One strong point of compositors is that you can fix the limitations of regular alpha blending (even including advanced things like linked lists OIT). It is also necessary if you wish to render special geometry. e.g. in FPS games the gun and hands should be rendered always in front of everything (i.e. never go "through" walls). Guns can have transparent submeshes. Compositors make this a breeze.

enum CompositorOpaquePassActionFlags {

-6. I don't even understand what this is supposed to mean. COPY_*_TO_BACKBUFFER sounds completely wrong.

There's no way to specify useful load/store actions (CLEAR is just one of them).
There's also no way to specify do not use actions (e.g. to instruct Godot that two consecutive passes are part of the same Vulkan RenderPass, which means under no circumstance we must not close the render pass between them; this is very important for TBDRs)

Godot renderer takes objects, renders them and that's it. To be able to implement most of these techniques, multiple, customizable opaque passes are required.

-7. IMO Ideally Godot should be able to render its internal stuff using the Compositor, and the user adds custom passes somewhere in the beginning, middle or end of the chain. What you're proposing is more like an optional components for a few fancy effects.

This would simplify the internal Godot architecture a lot.

[reduz]

@csubagio @darksylinc

I feel in the obligation to clarify that, this proposal is probably not what you think it is.

So to be in the same channel, I will explain what this is not:

This is not a flexible system to composite inputs and outputs of the rendering logic to be able to create complex post processing effects (such as AO, DOF, etc), where you connect input to outputs in a graph or something like that. The idea of doing this and providing it by default in Godot has been mostly dropped, as it's not fit for an engine like this. It's far too complex for the level of usability expected by most users of the engine.

So what is this then?

This is, primarily, a simple system for assigning materials to passes and giving them the ability to write to/ read from custom buffers, restricted to the opaque pass.

As I mentioned before, its for things like this:

• Fine grained stencil control (masking, bounding, and all the common stuff).
• Geometry decals
• Terrain blending to geometry.
• Ocean rendering and blending.
• Object outlining.
• Geometry blending to sky.
• Rendering information to custom buffers during rendering to then aid in custom post processing.
• Simple post processing effects.
• Many more things.

All simple things you can just develop easily, share and plug into your project without much hassle or understanding. how rendering works internally.

Users have been asking for a long time to be able to do all these things time and time again. The idea is that it can be provided in an extremely simple package where it is as easy as possible to write these shaders and integrating them to your project in a platform independent way and without having to get knee deep into how the actual compositing works.

Okay, then what if you do want to get knee deep into actual compositing and tweak rendering as much as you want?

In this case, this is what CompositorEffect is for. This lets you place callbacks into all stages of rendering code, and gives you access to all rendering data, including RenderingDevice. Just do whathever you want. If you want to create a graph based compositor, or override main rendering to do your own voxel rendering engine, you can do it. This is also renderer dependent, you get different buffers for deferred and forward as example, so up to you to deal with everything you need.

[PavelCibulka]

That idea sounds almost like Jason Booth "Better Shaders".

He has written compiler that takes parts (shader pieces) and compose, optimize and compile them into one shader. My explenation is probably butchering his work. You can look at his presentation.

https://www.youtube.com/watch?v=eEUT-7ObapQ&ab_channel=JasonBooth

[reduz]

@PavelCibulka Godot shaders are already like this by default, except far cleaner and user friendly. This just builds further upon it.

[csubagio]

@reduz I'm afraid you've misunderstood my comment: I'm not saying it should be more in the sense that it should be more complicated, I'm saying it should be less complicated, less of a unique special case, because it's actually something pretty fundamental. I believe your proposal, sitting alongside everything else already in the engine, looks like complication to me. It also looks like the basis for compounded future complication, which will come from incremental related community requests, based on similar evolution I've seen in other open and closed engines.

But, I'm not married to this at all. If this is what the community wishes, I'm happy to see it work. I'll also happily use it when it's appropriate!

[reduz]

@csubagio Complication depends on the point of view. As I said, the general philosophy behind Godot is always:

"Cater to the most common cases while leaving the door open for the corner ones (generally as lower level API)".

If its a common case, there will be an easy way to do it (the passes API in this proposal). If its a community request for something more complex, then this will not be part of the "common case" API, but can be done anyway using the low level one via CompositorEffect.

[nightrune]

Me and a few friends have been playing with voxel ray tracing. One of the tricks Teardown uses is a custom depth buffer they keep in memory between frames. This buffer gets moved based on velocity from the last frame. Then used as a depth prepass to stop ray tracing hidden objects i.e. Check depth to see if its in front of where you'd trace to anyway. If so, skip.

Would this proposal help that? Would you be able to keep the custom buffer between frames and update it with velocity?

Taken from this: https://acko.net/blog/teardown-frame-teardown/

[reduz]

@nightrune Yeah you can write all this logic in a single CompositorEffect on your own,

[BastiaanOlij]

@nightrune you can probably already see how far you can get with the current Rendering Effects implementation in #80214 which will be integrated into this proposal if it goes forward.

[Lerg]

This is a crucial feature for some games. As it allows to render stuff like no one else.
Unfortunately I'm not competent enough to provide concrete API suggestions but lately I've been working very close with the Sokol graphics library and I very much like its rendering pipeline API. Creating render passes and special effects like hdr or bloom is very easy and straightforward.

So I guess I can only recommend studying Sokol's API very close and bring the best from there. Here's for instance how to do render to texture pass in Sokol https://github.com/floooh/sokol-samples/blob/master/sapp/offscreen-sapp.c

[lrwillis]

Would the compositor support rendering directly in the native Graphics API?

I've been looking for a good way of adding a backdoor to the renderer to allow mixing in native Graphics API rendering. e.g. if using the OpenGL renderer, render some graphics directly in OpenGL, and mix that with the Godot rendered graphics.

Why a backdoor? To allow easier prototyping/mockups. e.g. You want to mockup what it would be like if you ported your game/app to Godot, but you don't want to have to port everything just for the mockup. This allows you to quickly port what's easy to port, and render the rest via the backdoor (using whatever existing codestack you have).

(Perhaps there's already a way to do that, and I just haven't found it)

[OhiraKyou]

Concerns

The following features are particularly important to me:

• Transparent blending (additive, usually) when rendering to a buffer, for stacking decals.
• Ability to render before opaque geometry (but after depth), to setup data for the opaque scene color to use.
• Support for floating point buffer color channels—rarely does buffer data represent just color.
• Compatibility renderer support.

Would those features, for the following use-case, be supported in the Compatibility renderer?

Use-case

My surface shaders (and, by extension, their materials) use multiple color palette textures whose x axis represent hue and y axis represent lightness. The UV coordinates they use to sample those textures can be modified by decals rendered to a palette offset effects buffer. Those coordinates can be negative and can go beyond the 0-1 range to support repeating palette textures (like a hue rainbow).

Rendering order

1. Opaque depth-only buffer
   • For decal masking.
   • Could also be merged into the next buffer as the alpha channel if float data is supported.
2. Opaque world space normals buffer
   • For the next buffer to use.
3. Color palette UV offset effect decals buffer.
   • This buffer acts as a modifier for the color palette UV coordinates of the primary opaque color pass. It enables local hue and lightness shifting that remains palette-based, regardless of which or how many color palettes the materials of the final opaque color pass use.
   • Requires additive blending to the buffer, for stacking UV offsets.
   • Requires support for float values that may be negative and/or outside of the 0-1 range (to support correct repeating texture sampling).
   • Samples depth for masking.
   • Some decals apply simple sphere gradients.
   • Some decals act like lights and limit their rendering to exposed surfaces by reading from the world normals buffer.
   • Some decals sample world normals to implement triplanar effects, like caustics, as palette coordinate offsets.
4. Opaque geometry (opaque scene color)
   • Some (but not all) shaders read from the UV modifiers buffer to shift their color palette sampling, resulting in changes to hue and lightness.
5. Transparent geometry
   • May sample opaque scene color.

[thygrrr]

This should most definitely not be limited to opaque passes only. It shows a limited understanding of what users (game devs) do in their transparent passes.

I also think instead of buffer, the term texture or target should be used, this is much more modern.

I think that instead of a graph, we need a stack of passes, including an "infinite" sandwich of implicit passes for the current material system with the "next pass" system.

But I'd rather like to see that replaced with a named pass system (passes are created based on actual use, e.g. if all my materials use 3 or less user passes, then there isn't a fourth pass).

A big important part of this feature is, and I think that it should be a STANDALONE proposal, is exposing CompareOperators and Stencil Ops to the material system. I'm working on a small engine change that does this, but the way the depth prepass is structured currently makes the straightforward solution incompatible with it.

[Calinou]

A big important part of this feature is, and I think that it should be a STANDALONE proposal, is exposing CompareOperators and Stencil Ops to the material system. I'm working on a small engine change that does this, but the way the depth prepass is structured currently makes the straightforward solution incompatible with it.

See #7174.

[QbieShay]

@reduz I'm missing options for non-opaque passes

[SlugFiller]

Okay, I was supposed to make a suggestion for this after some other people explained various issues regarding use cases and alternative renderers. But that didn't happen, so I'm just going to drop my suggestion here, for now, and let others explain the reasoning why it would be good later.

In the suggestions, passes are represented simply as integers.

void compositor_add_opaque_pass(RID p_compositor, int p_index);

The issue is that this limits passes to just be added at a specific point in the pipeline. While this might be good enough for most users, it might not be applicable to some use cases. (See comments above, and, hopefully, below as well)

At the same time, exposing the entire pipeline to the average user could easily become a footgun. If there are many insertion points, and the user can just as easily pick any, then the user can easily pick the wrong one. This could cause unexpected outcomes, and/or cause performance to tank.

The key is to, therefore, make it easy to add passes to the most desired insertion point - after the opaque pass, but make it unintuitive (but not impossible) to add them anywhere else. And the key to this, IMO, lies in a combination of named passes, and passes as RID instead of integers.

Here's what I have in mind:

// Query to allow determining the structure of the pipeline.
// Returns both built-in and custom passes.
// This allows having different behavior based on the type of pipeline. e.g. Add a pass at different points for forward vs deferred renderer.
// Does not give RIDs
bool compositor_is_pass_exists(RID p_compositor, StringName p_pass_name);

// FIXME: We do not provide a method for listing passes. This allows "hiding" passes the user doesn't know about, and doesn't need to know about. This reduces the chances of an inexperienced user shooting themselves in the foot
// We absolutely do NOT give RIDs in any situation. The built-in passes do not have associated RIDs at all, making them immutable.
//Vector<StringName> compositor_get_passes(RID p_compositor);

// The previous pass is intentionally specified as string, and not as RID.
// For built-in passes, the RID does not exist. Potentially, this could also be used by addons to add a pass after passes from the user, or from another addon, without needing to query for the RID.
// The default value directs users towards the only pass 99% of them should know about: The opaque pass
// This is also the only pass of which existence is guaranteed. Other passes may vary by renderer. They should still be documented, but in a place appropriate for "advanced options".
RID compositor_add_pass(RID p_compositor, StringName p_pass_name, StringName p_after_pass = "opaque");

// Takes an RID, and not a string. This means a user can only remove a pass they themselves added.
// They cannot touch built-in passes.
// Additionally, they can only touch a pass created by an addon, if said addon voluntarily exposes its RID.
void compositor_remove_pass(RID p_compositor, RID p_pass);

// All other methods accept `RID p_pass` instead of `int p_pass`. This ensures a user can only touch passes they've created, or have explicitly been given access to.
void compositor_set_opaque_pass_action_flags(RID p_compositor,RID p_pass,BitField<CompositorCustomBufferMask> p_flags);
void compositor_set_opaque_pass_stencil_clear_value(RID p_compositor,RID p_pass,int p_value);
void compositor_set_opaque_pass_depth_clear_value(RID p_compositor,RID p_pass,int p_value);
void compositor_set_opaque_pass_custom_buffer_usage(RID p_compositor,RID p_pass,BitField<OpaqueMultipassCustomBufferMask> p_usage);
void compositor_set_opaque_pass_custom_buffer_clear_color(RID p_compositor,RID p_pass,int p_buffer,Color p_color);
void compositor_set_opaque_pass_render_depth_prepass(RID p_compositor,RID p_pass,bool p_enable);

This allows maximum flexibility, while ensuring minimum footguns.

[OhiraKyou]

@SlugFiller (original comment)

the most desired insertion point - after the opaque pass

I disagree with the assumption that the most desired insertion point for a custom pass is after the opaque pass. Especially in forward rendering, the majority of your scene's color composition (e.g., applying custom decals, fake lights, and masks) is done in the opaque pass. Therefore, that's where you want most of your custom data ready for reading.

To generate that data, typically, a world normals and depth pre-pass, for masking and fading, are required. Custom passes follow those. After that, both opaque and transparent shaders can make use of the data. And, transparent shaders can combine that with the resulting opaque scene color.

My ideal order for most uses:

1. Pre-pass for opaque world normals and depth (needed for the next pass, which renders useful data).
2. Ideal insertion point, where masks and decals are rendered to generate data used for compositing the opaque color.
   • Custom decals
   • Custom lights (which the opaque pass will multiply with its unlit albedo)
   • Color modifiers
   • Volumetric masks and distance fields (e.g., underwater vs not, and/or water depth)
3. Opaque pass, where data is combined (or "composited") to render the final opaque scene color and, therefore, where it is needed the most.
4. Transparent (can still use data, like underwater masks, from custom passes).

The one issue you have to solve with this order is ensuring that shader-animated vertices render their post-animation world normals and depth in the pre-pass. In Unity, I got around this by animating vertices on the CPU using the jobs system so that my custom world normals and depth buffer shaders wouldn't have to account for animation. Incidentally, those buffers had to be custom due to lack of precision in Unity's existing, packed normals+depth buffer. But, if animated vertex data is captured correctly by the pre-pass (either by default or with multi-pass shaders) and made available with sufficient precision, custom world normal and depth buffers may not be necessary.

[astillich]

I'd like to second that inserting passes before the opaque pass is essential. A practical example: yesterday I developed a system for hiding walls obstructing the view on the player. The first idea I had was to render a sphere around the player into an alpha texture and then sample that texture when rendering walls, to make them gradually transparent around the player. This is not possible ATM, so I resorted to raycasts from the camera and making geometry invisible, which I think does not scale very well. It also doesn't look as good as a mask-based solution would.

[QbieShay]

I will note down here what i said in the meeting.

Since there are concerns for addon compatibility (what if i make a shader that needs to use a pass, and then the pass indices conflict) I thin Godot should have by default (of course editable) three opaque passes

• pre opaque
• opaque
• post opaque

This setup should cover most common usecases so that addon developers can easily ship their addons with some form of standard structure. Of course this should no prevent adding extra passes to the three I listed, but rather should give a "general standard structure" so that people that make addons can expect them to be plug and play in small/medium projects.

Note: I think depth and stencil should be copied at the end of each opaque pass and made available to the next pass. So pre-opaque will have depth&stencil from depth prepass, then it can write to depth and stencil, and then these changes are reflected in the opaque pass and so on.

[Ansraer]

Haha, sorry Slug, I fell straight into bed after the meeting yesterday and didn't have time to go voer my notes until now.

---

While talking about this proposal during the rendering meeting yesterday, we had two major concerns we kept coming back to. The first is that we are afraid that this proposal, in its current form, is too limited for many potential use cases.

Our understanding of your intentions reduz was that you only intend to allow the creation of additional opaque passes, so the overall rendering pipeline would look something like this:

1. Default opaque pass 0 (always exists, can't be removed or moved)
   1. pre-pass
   2. opaque
2. 0-n custom opaque passes, each with
   1. pre-pass X
   2. opaque X
3. Default transparent pass
4. Default Post processing

While this setup would be easy to implement and use, we all agreed that there are many valid use cases where you would want to place your custom pass somewhere else in the pipeline. As OhiraKyou just pointed out, it is essential to be able to run custom passes before the default pass if you wish to use the generated data in it. In addition, there are also reasons why you would want to have a custom opaque & transparent passes before/after transparent and post-processing.
QbieShay mentioned various VFX use cases during the meetings, while Slug, Apples and I brought up the rendering of FPS arms & weapons: They are usually comprised of several hero-quality assets that you want to render on top of everything else (including the transparent pass) with full support for lighting, transparent elements (e.g. a red dot sight) and particles (e.g. muzzle flashes, ...).
The current solution for rendering a player character for a first person game is to use Viewports, which has a number of well known problems.

Another concern that was raised is that interacting with custom passes and buffers based on their ID would result in conflicts. What happens when you add two add-ons to your project that each add multiple custom passes? Does pass #1 refer to the custom pass that was added by the first add-on or to the one added by the second add-on? What happens when both add-ons write data to custom buffer #1? As a possible solution for this, @SlugFiller proposed to use StringNames instead of indexes everywhere, so e.g. you could insert AnsraerPass after DefaultOpaque and write to AnsraerAlbedoBuffer. Later on you could read from AnsraerAlbedoBuffer without having to worry about another add-on overwriting the content of that buffer.

---

The second major discussion point during the rendering meeting was how this Compositor would interact with alternative renderers. I and several other meeting attendees were concerned that having a compositor that is designed around the current forward renderer could be restricting when we want to implement something else, such as e.g. a deferred renderer.

For effects that use different cameras, such as portals, the current design of completely independent passes would work fine (assuming we could move opaque passes before the default opaque pass):

1. Portal opaque pass 1
   1. depth pre-pass
   2. create g buffer
   3. lighting
2. Portal opaque pass 2
   1. depth pre-pass
   2. create g buffer
   3. lighting
3. Default opaque pass (always exists, can't be removed or moved)
   1. depth pre-pass
   2. create g buffer
   3. lighting

However, let's now consider an add-on that adds a custom opaque pass for terrain rendering, something that would be necessary if you want to do terrain blending as reduz has mentioned in the original post:

1. Terrain pass
   1. depth pre-pass
   2. create g buffer
   3. lighting
2. Default opaque pass (always exists, can't be removed or moved)
   1. depth pre-pass
   2. create g buffer
   3. lighting

Both the terrain pass and default opaque pass have the same camera, so using a fresh pre-pass for the second one is actively harmful. In addition, we just committed the cardinal sin of deferred rendering by running the lighting pass twice, something that defeats the entire point of a deferred renderer. A more reasonable rendering pipeline would be something along the lines of:

1. Depth pre-pass
   1. custom terrain
   2. default
2. Opaque Pass
   1. custom terrain
   2. default
3. Lighting

In the end, we agreed that it would be best to get some more user feedback, while also trying to come up with a more concrete idea as to what a deferred renderer in Godot would look like (@clayjohn volunteered to make a proposal for that when he has some spare time). We plan to come back to this proposal in a future meeting once we have a better idea of the requirements such a compositor needs to fulfill to be as useful for as many developers as possible.