Add per-light shadow cull masks to control which objects cast shadows

[mrjustaguy]

Describe the project you are working on

Open Environments

Describe the problem or limitation you are having in your project

I need to Specify which Lights cast Shadows from which objects, to optimize for performance, Currently only things I can do are Disable the Shadows from said geometry from every light, or none at all, which is bad in instances where for example, you know a Directional Light will always be in Shadow, but an Omni Light or Spot Light might not.

Currently Only Light has a cull mask, but that doesn't affect Shadows, just determines if the object is lit by the light (in 4.0 atleast, haven't tested in 3.x)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add Cull Mask for Shadows, This allows for cutting objects out of the Shadow Rendering Equation, for specific Lights (or telling the lights to render simplified shadows that might not work for every light casting shadows)

This can be used to greatly optimize Shadow Rendering in some scenes, without sacrificing Shadow Quality, at the expense of more artist time spent creating content and increasing performance, but doesn't hurt the workflow of those that don't invest the time to optimize like this.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Add Cull Mask to Shadows, just like there is a Cull Mask for calculating Light for object types.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, currently Shadow Casting is Binary, All Lights casting shadows go one of two routes: you cast, or you don't, for all lights lighting an object, no way to tell one group of lights to cast, and others not to cast for a given object.

Is there a reason why this should be core and not an add-on in the asset library?

Rendering is Core.

[Calinou]

Related to #2745.

This is a good idea overall, and it will come handy when using shadowmasking on low-end setups too (so you can enable shadows only for static objects).

Another benefit of having a shadow cull mask is that you can use a projector texture without actually casting any shadows (by disabling all shadow layers). This is useful to use a light with a projector texture without increasing the performance demands too much compared to a non-shadowed light.

For directional shadows, we may want to have more control over which splits an object will be included in. For instance, small fixtures don't need to be included in distant shadow splits as they will never cast noticeable shadows. This could perhaps be done automatically (e.g. using an object's AABB size in relation to the camera distance), but it may be too expensive to perform checks every frame for this.

[mrjustaguy]

For the directional Shadows, doing the automatic cull masking should be made an optional feature if it is implemented, as there could be multiple instances where this doesn't work properly, and it'll probably be difficult to make sure there aren't issues, which is why #3358 was closed as it was quickly finding conceptual issues poking holes all around the idea...

I mean it would have to be tested to see how much it'd act up in the real world, but I wouldn't get my hopes up.

[Lielay9]

Excuse my lack of knowledge but could this be used to "remove" self-shadowing without losing shadows from other casters? What I had in mind was to have a light on two layers, a foreground element on both layers, a middle-ground element visible on one with a shadow mesh on the other and a background element on both. Beautiful rendition in Paint:

Given that the middle-ground element (red torus) is visible only on the first layer, would it mean that it won't receive the shadows cast by the underlying shadow mesh on the second layer but still get shadowed by the foreground element¹?

Currently, to achieve the same effect I need to duplicate and render the scene twice and combine the result. I tried looking at the interwebs but the answer usually seems to be ranging from that it's either impossible(?) or can be done in 2 passes, so I'd be quite amazed if I could get the same effect on one. Nevertheless, this proposal would (probably?) at least make it possible to not have to duplicate the scene. At that point, it could even be fast enough to work for a little more complex game.

Footnotes

1. Non-self-shadowed objects couldn't receive shadows from others using the same technique unless sparing more layers. ↩

[Calinou]

This proposal is about shadow casting, not receiving. Disabling self-shadowing entirely requires control on which layers can receive shadows, and probably warrants a dedicated material flag that does not affect the ability to receive shadows from other objects.

[Lielay9]

This proposal is about shadow casting, not receiving. Disabling self-shadowing entirely requires control on which layers can receive shadows, and probably warrants a dedicated material flag that does not affect the ability to receive shadows from other objects.

Thanks for the clarification! So not quite what I was looking for. Regarding the latter, I doubt it is as easy as you make it sound 😓. My attempts were inspired by the recently openly released Goo-engine, a custom build of Blender. They had a quite neat set of features, being able to adjust scene and self-shadows (demonstration) per material, and having light/shadow groups (demonstration).

[jams3223]

This pull request might be useful.

[Vukbo]

Is there a milestone for this request? I stumbled into a problem where I would need such functionality. 🤔

[EMBYRDEV]

I might look at implementing this in the near future if there are no objections from the rendering team?

[EMBYRDEV]

Got this working in the GLES3 renderer already! It works in the Forward+ renderer too but there are some issues with the light clustering im working on fixing now.

Note how the red light is casting a shadow using the small cube but the white light is not.

[EMBYRDEV]

Working in Forward+ and Mobile!