Incorrect brightness when gl_FragColor is semi-transparent.

[bhouston]

Right now in the ThreeJS GLSL shaders, if there is a bright specular highlight on a transparent material, the highlight is not rendered correctly.

For example let say that at the end of the shader, the gl_FragColor = ( 8, 8, 8, 0.1 ). This should result in a bright highlight of let's say (0.8,0.8,0.8) being written to the frame buffer. But in fact, the values of gl_FragColor are clamped into the range of [0,1] before the value is passed to the gl.BlendFunc, thus even if you write gl_FragColor = (8,8,8, 0.1), it is actually equivalent to writing gl_FragColor = ( 1.0, 1.0. 1.0, 0.1 ), because the values will be clamped, which will result in a value of (0.1, 0.1, 0.1) being written to the frame buffer, much lower than the correct (0.8,0.8, 0.8) value.

I found a way to get around this artificial clamping of brightness on semi-transparent objects. The trick is to pre-multiply. Thus one would do this:

gl_FragColor = vec4( gl_FragColor.rgb * gl_FragColor.a, gl_FragColor.a )

And then change the blending modes for the material as follows:

    material.blending = THREE.CustomBlending;
    material.blendSrc = THREE.OneFactor;    // output of shader must be premultiplied
    material.blendDst = THREE.OneMinusSrcAlphaFactor;
    material.blendEquation = THREE.AddEquation;

The above settings do the standard blending but assumes a pre-multiplied gl_FragColor as input and it retained the whole range of acceptable intensities.

Here is a comparison of the improved results possible by using pre-multipled alpha in the gl_FragColor for high intensity semi-transparent materials -- this change is already live on Clara.io:

I am unsure if there are side effects to using pre-multiplied alpha. I suspect because the gl_FragColor.a is unchanged, there is few side effects.

[bhouston]

I can add this change to this PR here really easily: #5805

[crobi]

Just a side note: if you use light values outside of [0,1], shouldn't you be using a floating point render target and a HDR/tone-mapping post-processing pass? Otherwise colors of (8,8,8) and (1,1,1) will be the same white, which is not "correct"?

[bhouston]

@crobi You are correct that HDR would solve the issue as well, but an FP buffer and a separate pass is a lot more processing, and memory (at least 4x more memory for the frame buffer), which is costly on mobile. Such a change would still be fully compatible with HDR -- just set an FP frame buffer at the target -- but it also produces better results for semi-transparent materials in LDR at no real extra cost.

[crobi]

I can see that your approach solves your problem. I just found it strange to use light intensities of over 1.0 in LDR rendering. Is that a common thing to do?

I am unsure if there are side effects to using pre-multiplied alpha.

If three.js switches to using pre-multiplied alpha in general, code that uses custom shaders (partially constructed from the shader library) or custom blend modes might break.

[mrdoob]

Yeah, I can see this having side effects with other blending modes too?

[WestLangley]

@bhouston A valid premultiplied form must have the RGB components less than the alpha component, and ( 0.8, 0.8, 0.8, 0.1 ) is not a valid premultiplied color. I expect the consequences of invalid data are browser/OS-specific.

[crobi]

A valid premultiplied form must have the RGB components less than the alpha component

Source? I'm not a webgl expert, but I haven't seen this rule anywhere. The only place this could have an influence is if you export or import the entire context (as per here) - in this case the RGBA color could be de-multiplied and you'd get RGB component values of over 1. I can see this being problematic, but I haven't seen an actual rule for this.

[WestLangley]

@crobi Hmmm... I can't seem to find an "official" source, either. However, I believe I remember seeing @greggman provide a demo several years ago which showed that you can get unexpected results when blending, if a premultiplied form is expected, and the RGB components are greater than the alpha component.

In any event, this fiddle appears to show the same issue. If you change the clear color from ( 0.1, 0, 0, 0.1 ) to ( 1, 0, 0, 0.1 ), the black text on the canvas becomes an unexpectedly bright red, while the scene background color does not change.

[crobi]

the black text on the canvas becomes an unexpectedly bright red

Is that really unexpected though? A premultiplied (1, 0, 0, 0.1) is a super bright red of (10,0,0) with 10% opacity - and should result in a "regular" red of (1,0,0) when laid over a black background.

That being said, I still believe it is not a good idea to use HDR colors (with RGB components not limited to 1.0) with LDR buffers (where color components cannot exceed 1.0) - the presented approach just handles one special case when a color component is larger than 1.0 but smaller than 1/alpha.

[greggman]

Any values "out of range" are undefined. different browsers may have different results. Premultiplied of (1,0,0,0.1) is an out of range value if the context was created with premultipliedAlpha: true (the default).

[crobi]

Any values "out of range" are undefined

I can see the problems, I just don't see this defined in any specification. So I'm wondering whether this is an error/omission in the webgl spec, me being unable to find the part where it is defined, or just your conjecture? This is my interpretation:

• Within the OpenGL core, there is no such thing as "premultiplied alpha". This is just an interpretation of the RGBA values, and the GPU does not concern itself with that. The only limitation is that all RGBA components are clamped component-wise to [0..1] before written to an unsigned integer buffer. This clamping happens both at the end of a fragment shader and at the end of an alpha blending operation, so neither of them can generate "invalid" values.
• A WebGL context may need to "demultiply" RGBA values when interacting with the outside world (export the context to an image, blend the context with the rest of the HTML page). Since the spec does not talk about any restrictions, I would have assumed that as above, values are clamped to [0..1] before being written to a destination image.
• Only when blending the context with the rest of the page, I can see potential problems (the following is entirely my own speculation). If the browser uses premultiplied alpha internally, it will just alpha blend the context with the page. If the browser does not use premultiplied alpha, it might have to demultiply the values and then do alpha blending. The latter method will probably clamp the demultiplied values to [0..1] before blending, whereas the former method will probably just use the original values. The difference is basically what is shown in the image in the original post.

[bhouston]

Just to be clear that I didn't propose using premultipled Alpha frame
buffer. Just the output of the shader could be premultipled and if you use
the blend paramters I gave, it can be used with a standard
non-premultiplied buffer. Clara.io is already running with this method.
It works well with no side effects on any machines, because the change is
very local.

Best regards,
Ben Houston (Cell: 613-762-4113, Skype: ben.exocortex, Twitter:
@exocortexcom)
https://Clara.io - Online 3D Modeling and Rendering

On Fri, Dec 26, 2014 at 8:18 AM, Robert Autenrieth <notifications@github.com

wrote:

Any values "out of range" are undefined

I can see the problems, I just don't see this defined in any
specification. So I'm wondering whether this is an error/omission in the
webgl spec, me being unable to find the part where it is defined, or
just your conjecture? This is my interpretation:

• Within the OpenGL core, there is no such thing as "premultiplied
  alpha". This is just an interpretation of the RGBA values, and the GPU does
  not concern itself with that. The only limitation is that all RGBA
  components are clamped component-wise to [0..1] before written to an
  unsigned integer buffer. This clamping happens both at the end of a
  fragment shader and at the end of an alpha blending operation, so neither
  of them can generate "invalid" values.
• A WebGL context may need to "demultiply" RGBA values when
  interacting with the outside world (export the context to an image, blend
  the context with the rest of the HTML page). Since the spec does not talk
  about any restrictions, I would have assumed that as above, values are
  clamped to [0..1] before being written to a destination image.
• Only when blending the context with the rest of the page, I can see
  potential problems (the following is entirely my own speculation). If the
  browser uses premultiplied alpha internally, it will just alpha blend the
  context with the page. If the browser does not use premultiplied alpha, it
  might have to demultiply the values and then do alpha blending. The
  latter method will probably clamp the demultiplied values to [0..1] before
  blending, whereas the former method will probably just use the original
  values. The difference is basically what is shown in the image in the
  original post.

—
Reply to this email directly or view it on GitHub
#5810 (comment).

[WestLangley]

@bhouston wrote

Just to be clear that I didn't propose using premultiplied Alpha frame buffer.
Just the output of the shader could be premultiplied and if you use the blend parameters I gave, it can be used with a standard non-premultiplied buffer.

That is not quite true. In the general case, if the drawing buffer has an alpha channel not equal to 1, your proposed blending formula is the "mathematically correct" formula only if the drawing buffer and the shader output are both premultiplied.

Clara.io is already running with this method. It works well.

three.js supports a semi-transparent drawing buffer. Perhaps Clara.io is setting renderer.alpha = false, or the renderer's clear alpha to 1. three.js must work correctly in the general case, however.

[bhouston]

@WestLangley wrote:

That is not quite true. In the general case, if the drawing buffer has an alpha channel not equal to 1, your proposed blending formula is the "mathematically correct" formula only if the drawing buffer and the shader output are both premultiplied.

I'm not so sure about that.

three.js supports a semi-transparent drawing buffer. Perhaps Clara.io is setting renderer.alpha = false, or the renderer's clear alpha to 1. three.js must work correctly in the general case, however.

Clara.io is doing neither.

To understand why my proposal works, have a look at the standard blend function used:

    this.blendSrc = THREE.SrcAlphaFactor;
    this.blendDst = THREE.OneMinusSrcAlphaFactor;
    this.blendEquation = THREE.AddEquation;

This is what I am proposing to change it to:

    material.blendSrc = THREE.OneFactor;    // output of shader must be premultiplied
    material.blendDst = THREE.OneMinusSrcAlphaFactor;
    material.blendEquation = THREE.AddEquation;

Notice there is a single change in the blendSrc function. And instead of "SrcAlphaFactor" I am proposing doing that bit in the shader:

// line is the same as SrcAlphaFactor if it is the last line of the shader
gl_FragColor = vec4( gl_FragColor.rgb * gl_FragColor.a, gl_FragColor.a )

Thus this does not change how the background is handled in any way as compared to what was before. All I am doing is changing where the SrcAlphaFactor is being done to avoid an unnecessary clamp. That is it. Let me stress it is mathematically equivalent to what was being done previously, with the exception that gl_FragColor is no longer clamped to [0-1] prior to being multiplied by its alpha.

Now whether or not removing the artifacts the current approach creates are worth changing the blend mode used by the lit material (MeshLambertMaterial, MeshPhongMaterial) is worth it is unknown. (BTW I do not propose changing the default blend mode on all materials as it isn't necessary and it could cause problems with custom shaders.)

[bhouston]

For clarity purposes, I am referencing the default blend mode specified in Material here:

https://github.com/mrdoob/three.js/blob/master/src/materials/Material.js#L22

Which I undestand isn't used directly because BlendMode is set to Normal. But those lines are equilvaent to this line here in WebGLRenderer that sets the blending mode explicitly when it is Normal:

_gl.blendFuncSeparate( _gl.SRC_ALPHA, _gl.ONE_MINUS_SRC_ALPHA, _gl.ONE, _gl.ONE_MINUS_SRC_ALPHA );

https://github.com/mrdoob/three.js/blob/master/src/renderers/WebGLRenderer.js#L5681

[WestLangley]

@bhouston I hope you will take the time to understand what I have written, because what I am telling you is true.

The three.js NormalBlending mode is correct only if the drawing buffer is premultiplied and the shader output is not premultiplied.

This is why renderer.premultipliedAlpha is true by default, the NormalBlending mode is the default, and three.js shaders do not premultiply the RGB components on output.

Your proposed blending formula is the correct formula only if the drawing buffer and the shader output are both premultiplied.

[bhouston]

@WestLangley wrote:

Your proposed blending formula is the correct formula only if the drawing buffer and the shader output are both premultiplied.

Yes! I was trying to replicate NormalBlending mode while avoiding the clamp. So yes, it has the same limitations as NormalBlending mode. :)

I was not making a claim that this change is correct when compared to all blending modes, rather just Normal. Of course if you want a blending modes different than being equivalent to NormalBlending, you need to change things -- it is of course impossible to replicate different distinct multiple blending modes with a single equation. :)

So I think we are in agreement.

If @greggman concerns are true and that the value (0.8, 0.8, 0.8, 0.1) in the back buffer a real issue, then this isn't really possible. If it wasn't a concern, I would add a flag to materials called material.premultipledAlpha so one can retain this flexibility.

[bhouston]

@WestLangley would you support the addition to some materials, but probably not all, of a material.premultipledAlpha? Then I could use this mode in Clara.io without continuing to forking Three.JS but it doesn't force it on anyone else?

[WestLangley]

@bhouston

If you want to output from your shader so-called "valid" colors in premultiplied form, the way to do that is to create your custom shader material (perhaps by extending Phong), and set the material's blending mode to be the custom one you suggested.

On a related note, I am curious as to how you would answer the following questions. I am not sure there is an answer.

What color is represented by ( 1, 1, 1, 0.1 ) in premultiplied form?
What color is represented by ( 1, 1, 1, 0 ) in premultiplied form?

[bhouston]

@WestLangley asked:

If you want to output from your shader so-called "valid" colors in premultiplied form

They were never invalid. Just because a color, premultiplied or not premultipled, is not within LDR [0-1] doesn't make it invalid.

@bhouston asked:

@WestLangley would you support the addition to some materials, but probably not all, of a material.premultipledAlpha?

@WestLangley replied:

the way to do that is to create your custom shader material (perhaps by extending Phong),

So that is a no? :( I thought it would be a useful addition. Oh well. I can do it that way too.

@WestLangley asked:

On a related note, I am curious as to how you would answer the following questions. I am not sure there is an answer.
What color is represented by ( 1, 1, 1, 0.1 ) in premultiplied form?
What color is represented by ( 1, 1, 1, 0 ) in premultiplied form?

The first color (1, 1, 1, 0.1) is the premultipled version of (10,10,10, 0.1) -- one just has to solve the equation (ra, ga, ba, a) for (r, g, b, a), which is straightforward. Solve ra = 1, where a = 0.1 for r, thus r = 1/0.1, r = 10. It is a color outside of the LDR range, but when one considers transparency it is fully within the LDR range of [0-1]. My method preserved all colors that end up within the LDR range when incorporating transparency, where as without that change, transparent colors are unnecessary clipped.

The second color you show, (1, 1, 1, 0), is not a valid premultiplied color because there is no way to solve the equation (ra, ga, ba, a), because a is 0. Solving ra = 1, where a = 0 for r, leads to r = 1/0, which is undefined.

Does this make sense?

[WestLangley]

By "valid" I meant that the RGB components in a premultiplied representation were less than the alpha component. Shaders that output "valid" representations are fine, as long as the blending function is appropriately set.

I am still not sure about the consequences of blending premultiplied representations where the RGB components are greater than the alpha component. Consequently, I personally, would not do it.

I would like, at some point, to be able to respond more definitively to @crobi's comments.

[bhouston]

@WestLangley wrote:

By "valid" I meant that the RGB components in a premultiplied representation were less than the alpha component.

The values are just HDR premultiplied alpha, rather than LDR. Premultiplied HDR is used all over the place in the visual effects industry. It is a valid representation and is a straightforward extension of LDR premultiplied alpha.

@WestLangley

I am still not sure about the consequences of blending premultiplied representations where the RGB components are greater than the alpha component. Consequently, I personally, would not do it.

Well, with a non-FP framebuffer, you will run into clipping issues with HDR premultipled alpha but that is about it. Other than that, there should be no side effects as compared to LDR premultiplied alpha (where RGBA are all within the range [0-1].)

BTW I speak from experience, I wrote two HDR renderers that are widely used in the visual effects industry, and are of course used to create images that are incorporated into really complex compositing situations.

[WestLangley]

@bhouston Well, then maybe the reason I can't find a definitive problem with it is because there is no problem with it when used appropriately -- which is what you are saying... Although the fiddle I posted is still troubling me.

[bhouston]

@WestLangley Your example, as @crobi wrote, is behaving as expected. In premultipled alpha situation, the RGB of the foreground is added to the background. The white background (1,1,1) is multiplied by (1 - 0.1) to become (0.9, 0.9,0.9) and then the foreground red is added to get (1.9, 0.9, 0.9), but because the screen is LDR, you get (1.0, 0.9, 0.9). The black text starts with (0,0,0), which is multiplied by (1 - 0.1) to become (0,0,0) and then the foreground red is added to get (1.0, 0, 0). This is the expected result.

[bhouston]

@WestLangley If you come to the conclusion that HDR premultiplied alpha is not incorrect, it would still be very cool to have an option on shaders to output premultipled alpha results. I am sure there are applications outside of just my issue with unnecessary transparency clamping artifacts in LDR.

[greggman]

I don't know why I'm responding and I probably don't know all the issues but ...

It does seem like it would be nice if there was an option to have three.js render premultiplied alpha values given that's what the browser wants (in general) and matches the browser's use of .png images and the 2d canvas. The HTML spec assumes premultiplied alpha. WebGL added an exception (premultipledAlpha: false) because we knew the majority of older OpenGL apps (games) don't use premultiplied alpha.

In other words, if I want to render something over the page, like this doing the correct thing should be easier?

That seems pretty easy. Just like bhouston suggested add a material.premultipliedAlpha option that if true then when shaders are generated appends

gl_FragColor = vec4(gl_FragColor.rgb * gl_FragColor.a, gl_FragColor.a); 

That seems a lot nicer than requiring users to write all custom shaders if they want to use premultiplied alpha.

[WestLangley]

@greggman renderer.premultipliedAlpha is true by default, so the drawing buffer is premultiplied. The correct NormalBlending mode is the default, and the three.js shaders output non-premultiplied RGB components by default. With these settings, everything is internally consistent.

If @bhouston wants to output premultiplied values from his custom shaders, there is no problem doing that either. All he has to do is set the material blending properties appropriately. In fact, we could add a new blending option: THREE.PremultipyAlphaBlending, making the setting of the blending even easier. (I like that idea, actually.)

If in addition, @bhouston wants to output premultiplied values from some or all of the pre-defined three.js shaders (e.g., MeshPhongMaterial), then we would need to add the property material.premultipliedAlpha to achieve that. He would also have to set the blending mode to THREE.PremultipyAlphaBlending so things were consistent.

If we allow pre-defined shaders to output premultiplied values, it will impact post-processing logic, render-to-texture, and gamma correction, for example, but we will have to cross that bridge when we get to it.

[crobi]

Random remarks:

correct only if the drawing buffer is premultiplied

Are there really applications that use non-premultiplied drawing buffers? From what I've seen, we are talking about the following two options:

• Premultiply in the shader and use (ONE, ONE_MINUS_SRC_ALPHA) blending. The color is multiplied by alpha in the shader.
• Don't premultiply in the shader and use (SRC_ALPHA, ONE_MINUS_SRC_ALPHA) blending. The color is multiplied by alpha during the blending.

In both cases, a single full screen quad with saturated red color and 10% opacity on top of a black background would result in the drawing buffer containing (0.1,0,0) as RGB values. This is what you want the user to see - a very dark red screen.
AFAIK, there is no color-modifying operation in webgl that happens between fragment shader output and blending, so the only difference is whether the color gets clamped before or after multiplication by alpha.

Of course, you would see a difference if you use alpha values of <1, but disable blending. This could break some applications.

If we allow pre-defined shaders to output premultiplied values, it will impact post-processing logic, render-to-texture, and gamma correction

Hm, how?

leads to r = 1/0, which is undefined.

Shaders (and presumably GPUs in general) use IEEE 754 data types, so 1/0 is +infinity. I would assume that if you de-multiply (1,1,1,0), you get (+inf, +inf, +inf, 0), which gets clamped to (1,1,1,0) when written to a LDR buffer. I agree though that the de-multiplication should have been described in more detail in the spec.

[bhouston]

I'm proposed the material.premultipliedAlpha option in this PR: #8245