Make sure that semi-opaque objects are rendered correctly

[javagl]

(EDIT: Updated to fix a bug in the sample and the images)

Most of the information about how semi-opaque (or semi-transparent) objects should be rendered is encoded directly in the glTF asset. This information is contained in the technique.states and technique.states.functions. For example, the technique.states contains information about whether GL_BLEND is enabled or not. And technique.sates.functions contains information about the blend equation and blend functions.

(The only information that is not contained in the asset explicitly is the order in which the objects have to be rendered. Determining this order is far from trivial, and Order-Independent Transparency is an active research topic. For the simplest case, one has to make sure that opaque objects are rendered before semi-opaque objects are rendered. For the tests that are related to this issue, I implemented this locally in the jgltf-viewer, but these changes are not yet committed until this issue is resolved)

At the bottom of this issue, there is a glTF asset intended for testing the opacity handling.

The red triangle is rendered fully opaque, using the "opaqueTechnique".

The five blue triangles are rendered with different materials that use the "semiOpaqueTechnique":

• "semiOpaqueMaterial0" with an opacity of 0.0 (fully transparent)
• "semiOpaqueMaterial25" with an opacity of 0.25
• "semiOpaqueMaterial50" with an opacity of 0.5
• "semiOpaqueMaterial75" with an opacity of 0.75
• "semiOpaqueMaterial100" with an opacity of 1.0 (fully opaque)

By default, the technique.states.functions of the semiOpaqueTechnique contain the following settings:

"blendEquationSeparate": [ 32774, 32774 ],  // GL_FUNC_ADD, GL_FUNC_ADD
"blendFuncSeparate": [ 1, 771, 1, 771 ], // GL_ONE, GL_ONE_MINUS_SRC_ALPHA (2x)

The result is shown in this image:

The materials are ordered from upper left to lower right. So the upper left blue triangle is supposed to be fully transparent. Obviously, the result is utterly wrong.

Intuitively, I'd expect the values for the blend function to be

"blendFuncSeparate": [ 770, 771, 770, 771 ], // GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA (2x)

Because the pixels should be written with the source alpha, and the existing background pixels should be "blended out" or "overwritten" depending on the source alpha. The result looks correct for me:

I could now start trying different values for the RGB- and the alpha part, but I'm not sure whether "trial and error" will help here. The tool at http://www.andersriggelsen.dk/glblendfunc.php might be helpful to try things out, but I'm not sure how to systematically figure out the right settings, because I'm not entirely sure what the result is supposed to look like at all...

---

The glTF asset used for testing. The buffer data is embedded. The shaders are given below.

{
  "scenes" : {
    "scene0" : {
      "nodes" : [ "node0", "node1", "node2", "node3", "node4", "node5" ]
    }
  },
  "nodes" : {
    "node0" : {
      "translation" : [ -0.5, -0.5, -1.0 ],
      "meshes" : [ "meshWithOpaqueMaterial" ]
    },
    "node1" : {
      "translation" : [ -0.5, -0.1, -0.99 ],
      "meshes" : [ "meshWithSemiOpaqueMaterial0" ]
    },
    "node2" : {
      "translation" : [ -0.4, -0.2, -0.98 ],
      "meshes" : [ "meshWithSemiOpaqueMaterial25" ]
    },
    "node3" : {
      "translation" : [ -0.3, -0.3, -0.97 ],
      "meshes" : [ "meshWithSemiOpaqueMaterial50" ]
    },
    "node4" : {
      "translation" : [ -0.2, -0.4, -0.96 ],
      "meshes" : [ "meshWithSemiOpaqueMaterial75" ]
    },
    "node5" : {
      "translation" : [ -0.1, -0.5, -0.95 ],
      "meshes" : [ "meshWithSemiOpaqueMaterial100" ]
    }
  },
  
  "meshes" : {
    "meshWithOpaqueMaterial" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "opaqueMaterial"
      } ]
    },
    "meshWithSemiOpaqueMaterial0" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "semiOpaqueMaterial0"
      } ]
    },
    "meshWithSemiOpaqueMaterial25" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "semiOpaqueMaterial25"
      } ]
    },
    "meshWithSemiOpaqueMaterial50" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "semiOpaqueMaterial50"
      } ]
    },
    "meshWithSemiOpaqueMaterial75" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "semiOpaqueMaterial75"
      } ]
    },
    "meshWithSemiOpaqueMaterial100" : {
      "primitives" : [ {
        "attributes" : {
          "POSITION" : "positionsAccessor",
          "NORMAL" : "normalsAccessor",
          "TEXCOORD_0" : "texCoordsAccessor"
        },
        "indices" : "indicesAccessor",
        "material" : "semiOpaqueMaterial100"
      } ]
    }
  },

  "materials" : {
    "opaqueMaterial" : {
      "technique" : "opaqueTechnique",
      "values": {
        "diffuseParameter": [ 1.0, 0.0, 0.0, 1.0 ]
      }
    },
    "semiOpaqueMaterial0" : {
      "technique" : "semiOpaqueTechnique",
      "values": {
        "opacityParameter" : [ 0.0 ],
        "diffuseParameter": [ 0.0, 0.0, 1.0, 1.0 ]
      }
    },
    "semiOpaqueMaterial25" : {
      "technique" : "semiOpaqueTechnique",
      "values": {
        "opacityParameter" : [ 0.25 ],
        "diffuseParameter": [ 0.0, 0.0, 1.0, 1.0 ]
      }
    },
    "semiOpaqueMaterial50" : {
      "technique" : "semiOpaqueTechnique",
      "values": {
        "opacityParameter" : [ 0.50 ],
        "diffuseParameter": [ 0.0, 0.0, 1.0, 1.0 ]
      }
    },
    "semiOpaqueMaterial75" : {
      "technique" : "semiOpaqueTechnique",
      "values": {
        "opacityParameter" : [ 0.75 ],
        "diffuseParameter": [ 0.0, 0.0, 1.0, 1.0 ]
      }
    },
    "semiOpaqueMaterial100" : {
      "technique" : "semiOpaqueTechnique",
      "values": {
        "opacityParameter" : [ 1.0 ],
        "diffuseParameter": [ 0.0, 0.0, 1.0, 1.0 ]
      }
    }
  },
  "techniques": {
  
    "opaqueTechnique": {
      "program": "opacityProgram",
      "attributes": {
        "a_position": "positionParameter",
        "a_normal": "normalParameter"
      },
      "uniforms": {
        "u_modelViewMatrix": "modelViewMatrixParameter",
        "u_normalMatrix": "normalMatrixParameter",
        "u_projectionMatrix": "projectionMatrixParameter",
        "u_ambient": "ambientParameter",
        "u_diffuse": "diffuseParameter",
        "u_specular": "specularParameter",
        "u_shininess": "shininessParameter",
        "u_opacity": "opacityParameter"
      },
      "parameters": {
        "positionParameter" : {
          "type": 35665,
          "semantic": "POSITION"
        },
        "normalParameter" : {
          "type": 35665,
          "semantic": "NORMAL"
        },
        "modelViewMatrixParameter": {
          "type": 35676,
          "semantic": "MODELVIEW"
        },
        "normalMatrixParameter": {
          "type": 35675,
          "semantic": "MODELVIEWINVERSETRANSPOSE"
        },
        "projectionMatrixParameter": {
          "type": 35676,
          "semantic": "PROJECTION"
        },
        "ambientParameter": {
          "type": 35666,
          "value": [ 0.1, 0.1, 0.1, 1.0 ]
        },
        "diffuseParameter": {
          "type": 35666,
          "value": [ 1.0, 1.0, 1.0, 1.0 ]
        },
        "specularParameter": {
          "type": 35666,
          "value": [ 1.0, 1.0, 1.0, 1.0 ]
        },
        "shininessParameter": {
          "type": 5126,
          "value": [ 100.0 ]
        },
        "opacityParameter": {
          "type": 5126,
          "value": [ 1.0 ]
        }
      },
      "states": {
        "enable": [ 2929 ]
      }
    },
    
    "semiOpaqueTechnique": {
      "program": "opacityProgram",
      "attributes": {
        "a_position": "positionParameter",
        "a_normal": "normalParameter"
      },
      "uniforms": {
        "u_modelViewMatrix": "modelViewMatrixParameter",
        "u_normalMatrix": "normalMatrixParameter",
        "u_projectionMatrix": "projectionMatrixParameter",
        "u_ambient": "ambientParameter",
        "u_diffuse": "diffuseParameter",
        "u_specular": "specularParameter",
        "u_shininess": "shininessParameter",
        "u_opacity": "opacityParameter"
      },
      "parameters": {
        "positionParameter" : {
          "type": 35665,
          "semantic": "POSITION"
        },
        "normalParameter" : {
          "type": 35665,
          "semantic": "NORMAL"
        },
        "modelViewMatrixParameter": {
          "type": 35676,
          "semantic": "MODELVIEW"
        },
        "normalMatrixParameter": {
          "type": 35675,
          "semantic": "MODELVIEWINVERSETRANSPOSE"
        },
        "projectionMatrixParameter": {
          "type": 35676,
          "semantic": "PROJECTION"
        },
        "ambientParameter": {
          "type": 35666,
          "value": [ 0.1, 0.1, 0.1, 1.0 ]
        },
        "diffuseParameter": {
          "type": 35666,
          "value": [ 1.0, 1.0, 1.0, 1.0 ]
        },
        "specularParameter": {
          "type": 35666,
          "value": [ 1.0, 1.0, 1.0, 1.0 ]
        },
        "shininessParameter": {
          "type": 5126,
          "value": [ 100.0 ]
        },
        "opacityParameter": {
          "type": 5126,
          "value": [ 0.5 ]
        }
      },
      "states": {
        "enable": [ 2929, 3042 ],
        "functions": {
          "blendEquationSeparate": [ 32774, 32774 ],
          "blendFuncSeparate": [ 1, 771, 1, 771],
          "depthMask": [false]
        }
      }
    }
    
  },
  "programs": {
    "opacityProgram": {
      "vertexShader": "opacityVertexShader",
      "fragmentShader": "opacityFragmentShader",
      "attributes": [
        "a_position",
        "a_normal"
      ]
    }
  },
  "shaders": {
    "opacityVertexShader": {
      "type": 35633,
      "uri": "opacity.vert"
    },
    "opacityFragmentShader": {
      "type": 35632,
      "uri": "opacity.frag"
    }
  },

  "buffers" : {
    "buffer0" : {
      "uri" : "data:application/octet-stream;base64,AAABAAIAAAAAAAAAAAAAAAAAAACAPwAAAAAAAAAAAAAAAAAAgD8AAAAAAAAAAAAAAAAAAIA/AAAAAAAAAAAAAIA/AAAAAAAAAAAAAIA/AAAAAAAAAAAAAIA/AAAAAAAAAAAAAIA/",
      "byteLength" : 108
    }
  },
  "bufferViews" : {
    "indicesBufferView" : {
      "buffer" : "buffer0",
      "byteOffset" : 0,
      "byteLength" : 6,
      "target" : 34963
    },
    "attributesBufferView" : {
      "buffer" : "buffer0",
      "byteOffset" : 6,
      "byteLength" : 96,
      "target" : 34962
    }
  },
  "accessors" : {
    "indicesAccessor" : {
      "bufferView" : "indicesBufferView",
      "byteOffset" : 0,
      "componentType" : 5123,
      "count" : 3,
      "type" : "SCALAR",
      "max" : [ 2.0 ],
      "min" : [ 0.0 ]
    },
    "positionsAccessor" : {
      "bufferView" : "attributesBufferView",
      "byteOffset" : 0,
      "componentType" : 5126,
      "count" : 3,
      "type" : "VEC3",
      "max" : [ 1.0, 1.0, 0.0 ],
      "min" : [ 0.0, 0.0, 0.0 ]
    },
    "normalsAccessor" : {
      "bufferView" : "attributesBufferView",
      "byteOffset" : 36,
      "componentType" : 5126,
      "count" : 3,
      "type" : "VEC3",
      "max" : [ 0.0, 0.0, 1.0 ],
      "min" : [ 0.0, 0.0, 1.0 ]
    },
    "texCoordsAccessor" : {
      "bufferView" : "attributesBufferView",
      "byteOffset" : 72,
      "componentType" : 5126,
      "count" : 3,
      "type" : "VEC2",
      "max" : [ 1.0, 1.0 ],
      "min" : [ 0.0, 0.0 ]
    }
  },
  "asset" : {
    "version" : "1.1"
  }
}

The opacity.vert vertex shader:

#ifdef GL_ES
    precision highp float;
#endif

attribute vec3 a_position;
attribute vec3 a_normal;

uniform mat3 u_normalMatrix;
uniform mat4 u_modelViewMatrix;
uniform mat4 u_projectionMatrix;

varying vec3 v_position;
varying vec3 v_normal;

varying vec3 v_light0Direction;

void main(void) 
{
    vec4 pos = u_modelViewMatrix * vec4(a_position, 1.0);
    v_normal = u_normalMatrix * a_normal;
    v_position = pos.xyz;
    v_light0Direction = mat3(u_modelViewMatrix) * vec3(1.0, 1.0, 1.0);
    gl_Position = u_projectionMatrix * pos;
}

The opacity.frag fragment shader:

#ifdef GL_ES
    precision highp float;
#endif

varying vec3 v_position;
varying vec3 v_normal;

uniform vec4 u_ambient;
uniform vec4 u_diffuse;
uniform vec4 u_specular;
uniform float u_shininess;
uniform float u_opacity;

varying vec3 v_light0Direction;

void main(void) 
{
    vec3 normal = normalize(v_normal);
    vec4 color = vec4(0.0, 0.0, 0.0, 0.0);
    vec3 diffuseLight = vec3(0.0, 0.0, 0.0);
    vec3 lightColor = vec3(1.0, 1.0, 1.0);
    vec4 ambient = u_ambient;
    vec4 diffuse = u_diffuse;
    vec4 specular = u_specular;

    vec3 specularLight = vec3(0.0, 0.0, 0.0);
    {
        float specularIntensity = 0.0;
        float attenuation = 1.0;
        vec3 l = normalize(v_light0Direction);
        vec3 viewDir = -normalize(v_position);
        vec3 h = normalize(l+viewDir);
        specularIntensity = max(0.0, pow(max(dot(normal,h), 0.0) , u_shininess)) * attenuation;
        specularLight += lightColor * specularIntensity;
        diffuseLight += lightColor * max(dot(normal,l), 0.0) * attenuation;
    }
    specular.xyz *= specularLight;
    diffuse.xyz *= diffuseLight;
    color.xyz += ambient.xyz;
    color.xyz += diffuse.xyz;
    color.xyz += specular.xyz;
    color = vec4(color.rgb * diffuse.a, diffuse.a * u_opacity);
    gl_FragColor = color;
}

[javagl]

After investigating the rendering in Cesium, it seems like the rendering of semi-opaque objects in JglTF is "correct", as far as one can reasonably say this. Cesium uses Order-Independent Transparency to cause the objects to appear "nicely", but without OIT, they look exactly like the objects in JglTF. Particularly, when the input file says

"blendFuncSeparate": [ 1, 771, 1, 771 ], // GL_ONE, GL_ONE_MINUS_SRC_ALPHA (2x)

then the results in JglTF and Cesium look "equally wrong" (as in the first screenshot above). If the input file said

"blendFuncSeparate": [ 770, 771, 770, 771 ], // GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA (2x)

then the objects would appear more nicely, as in the second screenshot above.

The same applies to the "boxes" in the VC model, which have been discussed in KhronosGroup/glTF#576 : When disabling OIT in Cesium, then the boxes are visible there as well.

So the result is what the result is, according to the input file, and this issue can be closed for now.

[javagl]

The role of premultiplied alpha has to be investigated. In WebGL, it is ENabled by default. In Desktop OpenGL, it is DISabled by default. The colors that are delivered by the fragment shader have to be treated differently based on this setting. Since the fragment shader cannot be modified by the renderer, this may boil down to some (ugly, ugly) tweaks and workarounds of interpreting the technique.states.functions settings differently based on whether the asset declares to use premultiplied alpha or not. Right now, I think that implementing a "simple" glTF viewer may practically be impossible, but time will show.

[javagl]

I think that for a fragment shader that assumes premultiplied alpha, the computation at the end should be

gl_FragColor = vec4(color.rgb * color.a * u_opacity, color.a * u_opacity);

and not

gl_FragColor = vec4(color.rgb * color.a, color.a * u_opacity);

(because the u_opacity is basically part of the alpha component of the final color), but the glTF sample models are not doing this, so I have to assume that I'm wrong here.

[javagl]

After having read
http://webglfundamentals.org/webgl/lessons/webgl-and-alpha.html
https://developer.nvidia.com/content/alpha-blending-pre-or-not-pre
http://www.realtimerendering.com/blog/gpus-prefer-premultiplication/
http://tomforsyth1000.github.io/blog.wiki.html#[[Premultiplied%20alpha%20part%202]]
https://www.opengl.org/wiki/Blending#Colors
https://limnu.com/webgl-blending-youre-probably-wrong/
http://stackoverflow.com/questions/20362023/webgl-why-does-transparent-canvas-show-clearcolor-color-component-when-alpha-is
http://stackoverflow.com/questions/35372291/alpha-rendering-difference-between-opengl-and-webgl and
http://stackoverflow.com/questions/18783752/fragment-shader-and-coloring-a-texture
I assume that I can do either of the following:

• Use premultiplied alpha in the shader (assuming an opaque base color and a dedicated u_opacity uniform), by returning

  vec4 color = opaqueBaseColor;
  color.a = u_opacity;
  gl_FragColor = vec4(color.rgb * color.a, color.a);
  

  and

  glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
  

  to take into account that the source alpha was already multiplied in

OR

• Not use premultiplied alpha in the shader, by returning

  vec4 color = opaqueBaseColor;
  color.a = u_opacity;
  gl_FragColor = color;
  

  and

  glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
  

  (to let the blending take care of the alpha)

Both choices will not affect the renderer, but are solely determined by the input file. Specifically: The asset.premultipliedAlpha setting should not affect the rendering (except, maybe, for corner cases of transparent canvases).

---

Side note 1: The VC model uses

gl_FragColor = vec4(color.rgb * diffuse.a, diffuse.a * u_transparency);

in the shader, and I think that this does not properly take into account the u_transparency!

---

Side note 2: For the non-premultipled case, the blend functions may have to be

glBlendFuncSeparate(
    GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA, 
    GL_ONE,       GL_ONE_MINUS_SRC_ALPHA);

so that multiple semi-opaque objects are "stacked" properly - but this likely only makes sense when they are rendered in the right order, which cannot be guaranteed.

---

With both approaches sketched above, the results look "correct", as in the second screenshot.

(Corollary: The VC model contains an error. I'm open for arguments here)

[javagl]

The line that I currently consider to be "wrong" in the sense of an implementation of premultiplied alpha is probably caused by https://github.com/KhronosGroup/COLLADA2GLTF/blob/5cbbe4674c16aa05cc613d8b7c50f4557b5af611/shaders/commonProfileShaders.cpp#L1207