-
-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rendering Pipeline Improvements #82
Comments
Is it possible to create a single large vertex and fragment shader that accepts flags to turn on/off the individual ones? I understand that is a less optimized solution then merging mathematical operations to combine effects, but this would still drastically reduce CPU cycles from having to do multiple passes in the That might be a good stepping stone to get to a place where you can then introduce optimized merging of effects through the same API that poly-fills these later on?
Is that sort of what I'm describing? |
Yes, that's possible. There are two approaches to accomplish this inside a shader program:
The first approach may have a negative impact on the shader performance and should be avoided if possible. The second approach is practically the same as including or excluding the particular effect in the final shader. Changing a preprocessor variable entails a slow recompilation of the shader program.
Mathematical operations should always be merged manually on a per-shader basis as a general optimization step. But even if some shaders were programmed poorly, they may still get optimized by the WebGL shader compiler. Combining atomic operations from different effects could easily introduce bugs. Effects should be maintained as isolated routines, even after merging. CPU cycles will indeed be reduced, but that's just a minor side effect. The main focus of the proposed changes lies on the GPU workload, or rather the pixel throughput. While the complexity of the effect shaders will not be affected, the result of an effect will no longer be passed to the next one via a render target. It will instead be processed immediately by the next effect within the same shader program.
Actually, I don't think this library could be optimized any further when the proposed changes are in place. Right now, all effect passes take care of rendering by defining a complete shader material. They read texels from the input buffer and write new pixels to the output buffer. With the new approach, this data flow will be streamlined for maximum performance. Consider the following example: EffectPass ExampleWe want to implement a vignette effect and a noise effect in our application. import {
Clock,
PerspectiveCamera,
Scene,
WebGLRenderer
} from "three";
import {
EffectComposer,
EffectPass,
VignetteEffect,
NoiseEffect,
RenderPass
} from "postprocessing";
const composer = new EffectComposer(new WebGLRenderer());
composer.addPass(new RenderPass(new Scene(), new PerspectiveCamera()));
const clock = new Clock();
const vignetteEffect = new VignetteEffect();
const noiseEffect = new NoiseEffect();
// The EffectPass constructor accepts any number of effects.
const effectPass = new EffectPass(vignetteEffect, noiseEffect);
effectPass.renderToScreen = true;
composer.addPass(effectPass);
(function render() {
requestAnimationFrame(render);
composer.render(clock.getDelta());
}()); The Since effects will be required to define their main fragment and vertex shaders according to a specific standard, the Fragment Shader#include <common>
uniform sampler2D inputBuffer;
varying vec2 vUv;
INSERTION_POINT_HEAD
void main() {
INSERTION_POINT_MAIN_UV
vec4 inputColor = texture2D(inputBuffer, UV);
vec4 outputColor = vec4(0.0);
INSERTION_POINT_MAIN_IMAGE
gl_FragColor = outputColor;
} Vertex Shadervarying vec2 vUv;
INSERTION_POINT_HEAD
void main() {
vec4 outputPosition = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
vUv = uv;
INSERTION_POINT_MAIN_SUPPORT
gl_Position = outputPosition;
} Effect ShaderAn effect.frag// Custom uniforms and varyings.
uniform sampler2D bloomOverlay; // Example: bloom.
uniform float greyscaleIntensity; // Example: greyscale.
void mainUv(in vec2 uv, out vec2 transformedUv) {
// Effects that transform UV coordinates per fragment: glitch, pixelation.
}
void mainImage(in vec4 inputColor, in vec2 uv, out vec4 outputColor) {
// Example: greyscale.
outputColor = mix(inputColor, vec3(linearToRelativeLuminance(inputColor)), greyscaleIntensity);
// Example: bloom.
outputColor = texture2D(bloomOverlay, uv);
} effect.vert// Custom uniforms and varyings.
uniform vec2 texelSize;
varying vec2 vUv2;
void mainSupport() {
// Some passes may need to compute additional varyings.
vUv2 = uv.xyxy + texelSize.xyxy * vec4(-1.0, 0.0, 0.0, 1.0);
// However, most effects don't need to define a vertex shader at all.
} Functions, uniforms and varyings will be placed at the This is an initial draft which might change over time. I'm most concerned about how the glitch, pixelation and SMAA effects will affect each other. I'm sorry that this has turned into another wall of text. I hope I didn't miss the point of your question 😟 |
You very thoroughly answered my questions and your approach seems great. :) This is going to be a really cool enhancement. |
The changes outlined in this ticket have been implemented in |
Introduction
The purpose of the
postprocessing
library is to provide a package of well maintained and up-to-date filter effects for three.js. While there are already many improvements in place, the current code base of this library still roughly mirrors the postprocessing examples fromthree.js
and operates according to the following principles:EffectComposer
maintains a list of passes.This processing flow is quite simple and easy to use, but it's not very efficient.
Pipeline Problem
The
postprocessing
library currently operates in a wasteful manner when multiple passes are being used. The biggest issue that prevents optimal performance is that almost every pass performs at least one expensive fullscreen render operation. Using a variety of interesting filter effects comes at a price that doesn't justify the results.For the next major release v5.0.0 I'd like to replace the current naive approach with a more sophisticated one by introducing the concept of effects to improve performance and tighten some loose ends.
Merging Passes
The goal of most passes is to modify the input pixel data in some way. Instead of expecting each pass to write its result to a frame buffer just for the next pass to read, modify and write more pixels again, all of the work could be done in one go using a single fullscreen render operation. With the aim to improve performance, some passes already consist of multiple effects. When a pass is designed that way, it only needs to perform a single render operation, but it also becomes bloated and rigid. In terms of maintainability, cramming all the existing shaders into one is not exactly an option.
Still, there should be a mechanism that merges passes to minimize the amount of render operations globally.
Of course, there's more than one way to implement this feature, but passes should be merged whenever possible. This means that the user shouldn't be able to accidentally degrade performance. With this in mind, the merge process could either be automatic or explicit. In my opinion, it should be explicit to allow for more control. The convenience of a fully automatic merge mechanism is questionable because it could easily become unpredictable and difficult to handle due to its implicit nature. It goes without saying that the merge process should be as simple as possible and crystal clear.
EffectPass
At a closer look, passes can be divided into four groups. The first group consists of passes that render normal scenes like the
RenderPass
andMaskPass
. The second type doesn't render anything, but performs supporting operations. For example, theClearMaskPass
belongs to that group. Passes that render special textures make up the third group. GPGPU passes are a good example for this group. The fourth and most prominent group contains the fullscreen effect passes. Only this last group of passes can be merged.To devise a clean and efficient merge strategy, be it automatic or manual, these effect passes must be treated as a special case. Such passes actually contain effects and act only as carriers that eventually render them. Merging would only focus on the effects within the passes. Hence it makes sense to separate effects from passes. To further enhance the concept of effects, there should be only one pass that can handle effects.
In other words, there should be a monolithic
EffectPass
that is similar to theRenderPass
. While theRenderPass
can render anyScene
, theEffectPass
would be able to render anyEffect
and it would take care of merging multiple effects. This not only improves performance substantially but also reduces boilerplate code and guarantees an optimal processing order. For example, cinematic noise should not be applied before antialiasing. This approach is explicit in that the user adds effects to a specialized pass and expects it to organize them efficiently. It's also automatic in that the merging process itself remains hidden.Note that once a merge mechanism is in place, it will no longer be necessary to maintain bloated shaders. Complex passes can then safely be split into more fine-grained effects without losing the performance benefits of compound shaders.
Effects
The new
Effect
class will be very similar to thePass
class. AnEffect
must specify a fragment shader and a vertex shader according to a simple standard that will closely follow the one that Shadertoy uses. It may also declare customdefines
anduniforms
. Just like passes, effects may perform initialization tasks, react to render size changes and execute supporting render operations if needed but they don't have access to the output buffer and are not supposed to render to screen by themselves. Instead of arender
method, they would have something like anonBeforeRender
method.Consequences
Since the shaders from the chosen effects will be baked into a single shader, enabling or disabling one effect at runtime would require a slow recompilation of the shader program. As an alternative, the user can prepare multiple
EffectPass
instances with the desired effect combinations and enable or disable these passes as needed. This is also a reason why a purely automatic merge system would cause trouble.Besides the performance and coordination advantages, this new approach would allow every effect to choose its own blend mode from a list of built-in functions. An option to disable blending altogether would also be available to handle advanced compositing use cases in a very natural fashion.
Another interesting feature would be inter-effect-communication. Since effects will ultimately reside in the same shader program, they could easily declare output variables for other effects. For example, a chromatic abberation effect could optionally be influenced by a preceding bokeh effect.
One restriction that rarely becomes an issue is the limit on the amount of uniforms that can be used in a single shader program. With the monolithic
EffectPass
approach, this limit may be reached more quickly than usual. If that happens theEffectPass
will inform you about it and you'll either have to reduce the number of effects or use multipleEffectPass
instances. Most effects don't require additional varying fields so there shouldn't be an issue with that. However, some effects may use multiple varyings for UV-coordinates to optimize texel fetches. This case will be treated similarly and limitations will be reported to the user accordingly.Conclusion
At this point, it's hard to tell whether the presented changes can live up to the expectations. I did consider other options, but none of them looked as promising as the
EffectPass
approach.I'll try it out first to see if there are any hidden problems. In the meantime, feel free to share your thoughts.
✌️
The text was updated successfully, but these errors were encountered: