[Draft] WebGL2 Multisample Renderbuffers

[mattdesl]

This is an early draft of multisample RenderBuffers to ThreeJS. This is a WebGL2 feature.

Multisample renderbuffers don't even work yet in Chrome Canary or FireFox Nightly, but hopefully this can start the discussion.

I figure it should go like this:

renderTarget.samples = 4;    // 4x4 MSAA
renderTarget.samples = 0;    // disable MSAA
renderTarget.samples = null; // disable MSAA

Open questions:

• Is anything else needed to get this working?
• Where should we check for MAX_SAMPLES?
• Should this be a getter/setter to mark the render target as "dirty" and in need of update?

[bhouston]

I've written multiple before for OpenGL. Did you check to see which multi-sample levels are available before setting them? I think that is necessary for robustness.

[mattdesl]

@bhouston The spec already checks against MAX_SAMPLES and generates an INVALID_OPERATION error if above threshold. We could add that check for robustness and better error messaging before setting up the render buffer storage, throwing a new Error if it's above that value.

I'm not sure how to detect the possible sample levels. Seems like it needs to be power of two and based on the sized internal format of the render buffer.

See Table 2.nnn here:
https://www.opengl.org/registry/specs/ARB/framebuffer_object.txt

[mattdesl]

Ok, FireFox Nightly now supports multisample renderbuffers so we can start testing a bit.

Here is a nice easy reference:
http://ake.in.th/2013/04/02/offscreening-and-multisampling-with-opengl/

Notes:

• In WebGL2, glEnable( GL_MULTISAMPLE ) is probably replaced by { antialias: true } context attribute
• There is no glTexImage2DMultisample in WebGL2, I'm not sure if that's a problem
• You can't just sample a MSAA buffer from a shader, you first have to blit it to a 0-sampled FBO
• Your MSAA renderbuffer can't have a texture associated with it

The last two points introduces a bit of architectural changes. I'm not sure how best to implement with ThreeJS codebase, but I'll poke around.

[bhouston]

@mattdesl This is very nice work you are doing. :)

What we did to make our code simple is that we required people to create a texture for our MS render target, just like one creates a texture for non-MS render targets. But we didn't use it during MS rendering, but we used it as the write (not read) target for the MS render target (the 0-sampled FBO.) Thus we had a "commit"-type function that would read the MS render target to the texture.

[bhouston]

The benefit of the above design is that whether we did MS or non-MS, there was always a texture with the results of the render. Now we were not rendering to screen, which may introduce more design issues.

[mattdesl]

@bhouston That's nice, I think I'll do the same.

It looks like ThreeJS doesn't set up any color buffer render storage, which is needed for MSAA. I can implement a way that it only sets up specific color buffer render storage with MSAA targets, but it seems like it would be simpler to have all render targets specify a color buffer storage.

[mattdesl]

Hmm the only thing with your approach is that the sample counts need to match.

Say user creates a RenderTarget with samples=4, internally we need two WebGLRenderTargets; one with samples=4 and one with samples=0. Then we "commit" (blit) the framebuffers to make it all transparent to the user, so they are just operating on the internal samples=0 target. Is that correct?

[bhouston]

Yeah, you have the idea I was trying to communicate.

Also, I'm not saying this is the best idea, it is just what we did.

You could also make subclass of WebGLRenderTarget that contains the second MSAA one, and has a commit to the first. This way you can pass the MultiSampleWebGLRenderTarget into the code that uses just a WebGLRenderTarget and it will work. But that code could detect if it is a MSAA one (such as WebGLRenderer) and use the advanced features, commiting after it is done so that the class is still compatible with everything else.

[mattdesl]

Ok, now it looks like this:

    var target = new THREE.WebGLRenderTargetMultisample(window.innerWidth, window.innerHeight);
    target.texture.minFilter = THREE.LinearFilter;
    target.texture.format = THREE.RGBFormat;
    target.texture.generateMipmaps = false;
    target.stencilBuffer = false;
    target.depthBuffer = true;
    target.samples = 8;

If samples is zero, or if the context is WebGL1, the above has the same behaviour as normal WebGLRenderTarget.

It uses RGB8 or RGBA8 for internal formats (needs to match the 0-sample texture for blitting), so other ThreeJS texture formats won't work right now.

It will try to use min( target.samples, capabilities.maxSamples ) so technically you could set it to Infinity if you just want max samples. Instead, maybe we should just pass the exact user value (if > 0) to have WebGL throw errors? Not sure.

The code changes don't support MSAA cube targets. Maybe another design/architectural style would be needed for that?

In FireFox this seems to be working pretty well! I can get up to 8 samples on my late 2013 15" MBP, and performance is still pretty great. 🎉 Chrome is still bugging out, I've pinged Tojiro.

[bhouston]

It will try to use min( target.samples, capabilities.maxSamples ) so technically you could set it to Infinity if you just want max samples.

Well, one could add a true/false as to whether it should fail if it can not be achieved. I think in most cases one will want to have it use the highest samples it can successfully use, but in other cases, such as offline rendering, it may want to fall back to the manual MSAA pass that I PR'ed the other day.

[mattdesl]

Another thing that came up has to do with EffectComposer. Chances are you only want the first pass to use MSAA, and the rest should ping-pong two 0-sample targets. I ended up having to change some things in EffectComposer to get this working properly.

[bhouston]

It is easy, a pass can do anything if the result is a 0-sample target. I just created an MSAAPass that did this manually. Having a real Multisample pass is no different.

[MasterJames]

This seems informative and on topic?
https://www.cse.ust.hk/~psander/docs/tempcoj.pdf

What is the technical name for the method of aliases when a given pixel has samples taken randomly from four surrounding quadrants, and if they are close enough (agreeing) does not sample 4 more in that disagreeing quadrant? Eventually when they are all close enough (to a set limit) an appropriate relative average is made? Is this exclusive to the definition of Raytracing (of which there are many formulas Hanning, blackman, catmull-rom)? is it called Ray Variance?

So the idea of that here is not a sensible approach?

I guess the example is in the middle of a sphere it does less sampling as it's more uniform, while the edge requires more samples. (Another form of the formulas simply called [in Houdini etc.] "Edge Detection Filter" comes to mind in this example.)

Is this what you mean by "real Multisample pass"? and again I'm guessing it's not GPU or real-time friendly but it sounds maybe more efficient if possible (on mass in the GPU).
And thanks in general for taking the time/years to help me understand the threeJS issues (so I'm more useful here in the future), as well as more specifically here (this issue) to grasp where the limits are when dealing with GPU real-time rendering compared with raytracing as it sounds like you're doing a less efficient but similar things.

The article also talks about using samples over time somehow, but that suggests pixel tracking vectors to my thinking?

This works well and looks pro and proper (nice job).

Further I keep thinking about how a helper could scan a scene to ultimately disable quality features, based on (adjustable user/device preference settings like) framerate (and/or object distance) and this seems like the perfect thing for dynamically changing based on performance on the specific device. Ironically you maybe need aliasing more at distance, where as geo-detail (and/or particle count) might work the other way around [similarly like I figure shadow-map res and probably most textures would]. You could even disable shadows entirely for distance objects if you preferred object detail etc.

[MasterJames]

I guess this is essential an Effect Composer, Vector based Motion Blur Anti-aliasing proposal (at least to me it seems to be) suggested, I believe by this same article linked to earlier https://www.cse.ust.hk/~psander/docs/tempcoj.pdf in my previous message, and am still wondering if I’ll finish reading it myself before writing on it?!
After some grazing on it and thinking, it seemed to me it would be possible that thinking temporally with only a start and end frame within a second, a frame in between can be estimated. This means with an animated spline description of a pixel position over time that is computed from motion tracking vectors, basically used for motion blur as only a frame by frame vector, is instead used to compute a curve.
https://www.youtube.com/watch?v=zNiI8hbYbxQ
It’s not just “pixel vectors” but you use that to generate a spline which I believe is a good (linear algebra ~ matrix = solve ~ GPU) GPU kind of problem. A probability curve is generated. On average this is smaller as they also work together like voxels, which exhibit physics type properties that also look like a GPU problem.
With this probability curve an extra accurate component can be incorporated into an anti-aliasing pipeline.
Its contribution may not be worth the effort to get the job done. Still I imagine it’s a temporal way of describing a sequence of frames (not just as help with anti-aliasing), so that any client can compute whatever number of frames they want or at least where the detail is needed (at distance is typically) most useful and slow to change.
Noted: is that we also have the advantage of knowing slightly ahead of time what the animation is already going to be, not only predicatively but in certain cases matter-of-factly, and in real-time that’s probably helpful information.
Using the computed vectors of the existing model animation suggests perfect knowledge of this, so ya it’s just adding another way of adding a (temporal) dimension to your anti-aliasing compute on a pixels/sec level.

The color itself is also a spline vector.
Flicker detection equals sample depth.

[mrdoob]

Aesthetics note... Could we name it WebGLMultisampleRenderTarget instead? 😇

[mattdesl]

Ok, renamed! 😄

Outstanding issues:

• How important is MSAA cube render targets?
• How should we handle renderTarget.samples when it's over the maximum capabilities? Is the current clamp to max approach OK?
• This needs a demo added to examples/

[mattdesl]

@bhouston I can't seem to figure out how to get a MSAAPass without an extra (unnecessary) pass, since each pass can't modify the read/write targets of the EffectComposer. e.g.

function MSAAPass (scene, camera) {
  this.scene = scene;
  this.camera = camera;
  this.enabled = true;
  this.clear = true;
  this.needsSwap = false;
}

MSAAPass.prototype = {

  _updateMultisampleTarget: function (width, height) { ... },

  render: function (renderer, writeBuffer, readBuffer, delta) {
    // Lazily create / update a WebGLMultisampleRenderTarget
    this._updateMultisampleTarget(readBuffer.width, readBuffer.height);

    // ???
    // renderer.render(this.scene, this.camera, readBuffer, this.clear);
  }
};

[bhouston]

@mattdesl asked:

How important is MSAA cube render targets?

I am in favor of the basics first. :) More features can come later.

How should we handle renderTarget.samples when it's over the maximum capabilities? Is the current clamp to max approach OK?

Yes, we should clamp to max in my opinion. No crashes.

This needs a demo added to examples/

Very much so. Why not just take my ManualMSAARenderPass example and swap out my code with yours -- you should have similar results but it should be faster. :)

[mrdoob]

#9965