Compositing
I'll just copy this from the readme for now.
There's three different ways to do compositing, based on if you're using just the Optimised texture, the Foreground and Background textures, or all three.
This is the setup you'll probably be using. An extra camera is used to render the scene two additional times. The foreground texture can then be used as a premultiplied image, so that emissive (glowing) objects and fog will correctly appear in front of the model.
There are some extra steps needed for limit the influence of the foreground, so that areas of the background not covered by your model don't experience doubled glow. That can be done either with an Advanced Mask filter or some adjustments to blend modes.
Uses Plugin, Uses Groups
To composite this config:
Uses Plugin, No Groups
To composite this config:
No Plugin, No Groups
To composite this config:
If there are no emissive objects between you and the camera, this is the way to go. The Optimised texture has the RGB channels of a background texture, but with the Alpha channel of a foreground texture. It only adds one render, and is therefore quite efficient.
The OBS layout for compositing Optimised configs is very simple.
That's it. You don't even need any filters.
This config is not very efficient. You're rendering the active scene three extra times. Avoid, if you can.
In Beat Saber, the alpha channel is overwritten during post-processing. For reasons unknown, Beat Saber is missing the code needed to save and re-apply the original alpha channel at the end of the frame, so our Foreground render does not not produce a usable premultiplied texture. Luckily, the Optimised texture still works as intended, and can be used to fix the problem.
To composite this config:
You only need a single filter here. The Foreground source must be masked to the Alpha channel of the VNyan source, so that foreground objects don't have double-strength glow.
My settings for that mask are this:
If you don't know what premultiplication is, watch this video. It's about compositing in general, but does explain premultiplication, as part of that.
OBS seems to prefer straight alpha. OBS' plugin API gives direct enough control for a plugin source to set blending correctly for premultiplication, but adding almost any effect filter will set it back to straight alpha. That means that things like glow and fog will disappear.
I've seen this with an empty shaderfilter, the render delay filter, and colour correction.
If you need to apply a filter to a premultiplied source, either find a way to apply it to a group further up, or separate the source into two with different blend modes.