Fast Polygon Lights (would include Area)

[bhouston]

Description of the problem

Some new SIGGRAPH research results. A fast way to do polygonal lights, which would be a generalization of area lights. I find the results compelling. We should have area light support in ThreeJS and this would be an effective way of doing it:

https://eheitzresearch.wordpress.com/415-2/

Three.js version

• Dev
• r76
• ...

Browser

• All of them
• Chrome
• Firefox
• Internet Explorer

OS

• All of them
• Windows
• Linux
• Android
• IOS

Hardware Requirements (graphics card, VR Device, ...)

[bhouston]

Sample WebGL code here:

http://blog.selfshadow.com/sandbox/ltc.html

[MasterJames]

Awesome! Where can I learn about what Skew is more? It seems like depth could skew the normals or something to be even more accurate.
Of course the example pops a GL error on mobile :-( but... OMG OMG +++++1 !

[carstenschwede]

@bhouston This looks great! Does the link work for you? http://blog.selfshadow.com/sandbox/ltc.html crashes on my OS X, GTX 770 with
ltc.html:598 Uncaught GL Error: 37442 CONTEXT_LOST_WEBGL
no matter the browser [Safari (Recent), Chrome (M50 + Canary), Firefox (Recent + Nightly)].

[abelnation]

I'm interested in this, especially on the math lib side of things. From perusing the paper, it seems like a key part of making it work is to precompute the parameters of the different cosine distributions based on the various angles.

Do you foresee this as:
a) only having to be done once, saved as a texture in three.js, and then just leveraged by shaders?
b) having to be computed on the fly and stored in a texture?
c) being able to compute everything needed in a shader?

[abelnation]

Had some fun sketching out a LinearlyTransformableSphericalDistribution class in code:

[bhouston]

There is example code for this in the link above. :) I'd start with on the fly computation

If you contribute this to ThreeJS would be awesome to have reusable glsl math structs and functions to go with them. Hugely useful.

I think that lights very very rarely change shape, but they often move around. That determines the caching strategy you can use. I think that for mobile you really do want to cache what you can because glsl compute shader capacity is greatly limited.

Most area lights are circular or rectangular. Other shapes are incredibly rare. Thus even if this has great flexibility for arbitrary shapes, for the most part that isn't required.

[bhouston]

BTW Abelnation, IES lights use a spherical distribution as well, but they are pre-computed and uploaded to WebGL as a texture that one then samples them based on one's relative position from the light: https://docs.unrealengine.com/latest/INT/Engine/Rendering/LightingAndShadows/IESLightProfiles/

[abelnation]

Okee, I'm spending some time on background research to make sure i have an intuitive sense for the math involved in the relevant papers before I jump into playing with code. Perhaps we can sync up and discuss implementation ideas once i've sunk my teeth into the various concepts.

[abelnation]

@bhouston is there any process or place to write up something like planning/design documents to discuss new interfaces etc.? In doing my research, and thinking about the code-base, this definitely sounds like a feature that will have a number of smaller components, e.g. re-usable shader utils, a generic AreaLight class, updates to the WebGLRenderer, handling shadows, etc.

I'm happy to discuss further.

[abelnation]

Hey @bhouston, fyi, I'm back from some traveling and jumping into this full-tilt. I have a pretty good sense for the steps to get this implemented.

Sometime in between now and my last comment, it seems the authors of the paper did a significant, and much needed cleanup pass on their sample code, removing a number of confusing and unused methods and structs.

Let me know if you'd like to sync up on this, but I'll be cranking on getting an example page for AreaLights setup. Will try to get my repo deployed to the web so I can post links for work in progress.

[abelnation]

Ok. Current skeleton example is deployed here:
http://groovemechanic.net/three.js.release/examples/#webgl_lights_rectarealight

Note: area light calculation is not yet in place. it's currently using a directional light as the actual lighting. This will be my sandbox example I'll be developing/testing within.

[mrdoob]

ERROR: 0:90: 'MAX_NUM_POLYGON_POINTS' : undeclared identifier 
ERROR: 0:90: '' : constant expression required 
ERROR: 0:90: '' : array size must be a constant integer expression 
ERROR: 0:96: 'struct' : syntax error 

WIP I guess?

[abelnation]

@mrdoob very much so. will holler when i have something less WIP

[abelnation]

ok, skeleton placeholder should be compiling now, but still without the AreaLight calculation. It's currently using point light code as a stub implementation:
http://groovemechanic.net/three.js.release/examples/#webgl_lights_rectarealight

@bhouston would be great to get your thoughts on this. i'm currently working to integrate the LTC light calculation into the blinnphong material shader model.

Currently, the light calculation model used in lights_template appears to use a single irradiance vector value to calculate the final diffuse and specular terms. For AreaLights, unfortunately, the Irradiance cannot be reduced down to a single irradiance vector to calculate the specular term. Instead, radiance is calculated for each edge of the polygon, using different BRDF parameters for each point.

Given that, I'm looking at how the material model might be augmented. It seems like a 3rd Render Equation (RE) function would be added, e.g. RE_Area and RE_Area_BlinnPhong which, instead of calculating a single IncidentLight value for the AreaLight type, it will have to take each edge point of the polygon into account, using different BRDF parameters for each point.

Thoughts?

relevant files I'm currently with:

• lights_template.glsl
• lights_pars.glsl
• lights_phong_pars_fragment.glsl

[bhouston]

In this code, where is the RE stored? I believe it isn't part of the integrations of the edges, rather it is part of the matrices passed into the integration as well as the texture lookup? I wonder if it is possible to separate the RE from the integration itself. Thus one would have a getAreaLightIntegrals() function of some type, similar to the other light queries, and it would take the geometry and the area light struct, that would return data that could then be used in the RE. So that the RE is easy to switch between GGX and BlinnPhong.

The RE could have a helper that can be used to pass in data to the getAreaLightIntegrals(), like we have for the IBL stuff (indirect specular, indirect diffuse maps):

https://github.com/mrdoob/three.js/blob/master/src/renderers/shaders/ShaderChunk/lights_physical_pars_fragment.glsl#L47

https://github.com/mrdoob/three.js/blob/master/src/renderers/shaders/ShaderChunk/lights_template.glsl#L126

But I haven't studied this to a large extent thus I am unsure if this division / design is possible. I would love to be able to share most of your code between GGX and BlinnPhong in some way.

I'd call the lights PolygonAreaLights for the time being, to differentiate them from other light types.

I'd also only make them one-sided. The two sided case is increably rare and we can just add two lights to a scene for that case.

[bhouston]

Actually we could just implement the BlinnPhong RE as we sort of did for the specular environment map, and all we did was convert from GGX roughness to BlinnPhong shininess in a way that made the results nearly equivalent. We could do that for supporting both the GGX and BlinnPhong model here -- use the same complex RE, but pass in different parameters to it.

[abelnation]

well this is definitely progress :)

http://groovemechanic.net/three.js.release/examples/webgl_lights_rectarealight.html

[bhouston]

beautiful. CAn you make a specular reflection as well on the ground surface?

[abelnation]

yeah, currently the specular term seems to be evaluating to zero. still have some debugging to do

[abelnation]

i suspect the issue is a problem calculating the proper view angle from the camera to the point on the surface being shaded in the fragment shader.

Am I correct in understanding that the GeometricContext is fully transformed by the modelViewProjection matrix? Currently, to calculate the incident view angle, I am doing:

// Calculate angle between surface normal and view direction in radians
float theta = acos( dot( geometry.normal, geometry.viewDir ) );

Anything wrong w/ that calculation that you can see?

[WestLangley]

Am I correct in understanding that the GeometricContext is fully transformed by the modelViewProjection matrix?

The modelViewMatrix. The GeometricContext is in camera space.

Your formula looks OK.

[abelnation]

Update:
I believe I've verified that the angle calculation is correct. Looking into whether I've loaded the precomputed ltc values texture data properly into the shader.

[abelnation]

Anyone see anything wrong with these snippets demonstrating how I'm loading precomputed values into the shader? As far as I can tell, I'm getting all zero values when loading values from the texture. Let me know if you need more context.

UniformsLib.js

THREE.UniformsLib.LTC_MAT = new Float32Array( [ ... lots of precomputed values (4 x 64 x 64) ... ]);
THREE.UniformsLib.LTC_MAG = new Float32Array( [ ... lots of precomputed values (1 x 64 x 64) ... ]);

THREE.UniformsLib.LTC_MAT_TEXTURE = new THREE.DataTexture(
        THREE.UniformsLib.LTC_MAT,
        64, 64, THREE.RGBAFormat, THREE.FloatType,
        THREE.UVMapping,
        THREE.ClampToEdgeWrapping, THREE.ClampToEdgeWrapping,
        THREE.LinearFilter, THREE.LinearFilter, 1);

THREE.UniformsLib.LTC_MAG_TEXTURE = new THREE.DataTexture(
        THREE.UniformsLib.LTC_MAG,
        64, 64, THREE.AlphaFormat, THREE.FloatType,
        THREE.UVMapping,
        THREE.ClampToEdgeWrapping, THREE.ClampToEdgeWrapping,
        THREE.LinearFilter, THREE.LinearFilter, 1);

Object.assign(THREE.UniformsLib, {
...
    ltc_brdf: {
        "ltcMat": { type: "t", value: THREE.UniformsLib.LTC_MAT_TEXTURE },
        "ltcMag": { type: "t", value: THREE.UniformsLib.LTC_MAG_TEXTURE }
    },
...
});

ShaderLib.js

    'phong': {

        uniforms: THREE.UniformsUtils.merge( [
            ...
            THREE.UniformsLib[ 'ltc_brdf' ],
            THREE.UniformsLib[ 'lights' ],
            ...

        ] ),
        ...
    },

bsdfs.glsl

// function to calculate texture coordinate for precomputed value texture
vec2 ltcTextureCoords( const in GeometricContext geometry, const in float roughness ) {

    vec3 N = geometry.normal;
    vec3 V = geometry.viewDir;
    vec3 P = geometry.position;

    // view angle on surface determines which LTC BRDF values we use
    float theta = acos( dot( N, V ) );

    // Parameterization of texture:
    // sqrt(roughness) -> [0,1]
    // theta -> [0, PI/2]
    vec2 uv = vec2(
        sqrt( saturate( roughness ) ),
        saturate( theta / ( 0.5 * PI ) ) );

    return uv;

}

...

    // load data using texture coordinate
    vec4 brdfLtcApproxParams = texture2D( ltcMat, uv );
    float brdfScalar = texture2D( ltcMag, uv );

[WestLangley]

@abelnation

1. I am a bit surprised you are working with Phong material, as the paper is based on using LTCs to approximate the GGX distribution. I think you should start with MeshStandardMaterial which is GGX-based.
2. If I remember correctly (please check), the coordinate system used in the paper is different from the three.js right-handed system with Y-up. That means you may have to return to first principles and re-derive the math in the paper. I would conduct a separate experiment to demonstrate you are approximating the BRDF correctly before using the approximate BRDF in the rendering equation.
3. The LTC approximation is indexed by roughness and the cosine of the angle -- not the angle itself.

[abelnation]

@WestLangley

My previous question is whether or not I am loading texture data correctly as per my code snippets above. All debugging w/ the shader points to the texture data not being loaded at all.

To address you points:

RE 1) As I was operating under little to no guidance, I just jumped into the Phong model as a place to get started. My goal was to get the math and a proof of concept working, and then integrate it in all the proper places. I've pretty much subverted most of the normal phong equation by writing a new Render equation for area lights and simply convert the blinn shininess to ggx roughness before looking up texture values. Regardless, I intend to clean it up and integrate it with the standard model when all's working as expected.

RE 2) I am working with the exact brdf fitted values that are used in the webgl demo linked by the paper. I do not intend to nor really have a good idea how to re-fit the approximation for a different coordinate system.

RE 3) In the webgl demo they link to, the precomputed data is parameterized by theta, not cos(theta)

        float theta = acos(dot(N, V));
        vec2 uv = vec2(roughness, theta/(0.5*pi));
        uv = uv*LUT_SCALE + LUT_BIAS;

        vec4 t = texture2D(ltc_mat, uv);

[WestLangley]

@abelnation Sorry if I was incorrect regarding point (3). I still stand by my recommendations (1) and (2), however.

[MasterJames]

Is that working on mobile for anyone? It doesn't for me. If it doesn't work on mobile I'll never use it or it needs to detect the error it triggers and have a fallback to 4 corner point lights or something maybe?

[mrdoob]

If it doesn't work on mobile I'll never use it or it needs to detect the error it triggers and have a fallback to 4 corner point lights or something maybe?

Relax... 😉

[abelnation]

@MasterJames slow your roll. the posted link is my dev version, which is completely broken right now

[abelnation]

RE my use of DataTexture, I would still love some feedback on whether I'm using it correctly to load a Float32Array into the shader

[abelnation]

Hmm, for some reason, setting needsUpdate = true is not working. I also tried explicitly setting the version number to a non-zero value (which is what happens when you set needsUpdate = true) to force the texture upload, and the version is somewhere set back to zero. If set breakpoints and explicitly set the version back to 1, it will successfully upload the texture date. Any ideas what might be causing the texture version to revert back to zero?

[abelnation]

Hmm, figured it out. The DataTexture I was creating is somewhere copied by WebGLRenderer before it is used. Currently, Texture.copy does not copy the version number over. Therefore, the renderer was seeing my Texture with a version number of zero, despite me setting the needsUpdate = true on the original.

[abelnation]

Ah, much better.

[abelnation]

I've created a separate place for me to deploy my example page that won't get updated with my debugging efforts:

http://groovemechanic.net/three.js.release/examples/#webgl_lights_rectarealight

[mrdoob]

The DataTexture I was creating is somewhere copied by WebGLRenderer before it is used.

That sounds like a bug?

[abelnation]

@mrdoob which is the bug? the fact that the Texture object gets copied? or that needsUpdate is not properly set on the receiver? I do believe that the latter is a bug, and easily fixed. I don't see much justification not to copy that value over.

[mrdoob]

@mrdoob which is the bug? the fact that the Texture object gets copied? or that needsUpdate is not properly set on the receiver? I do believe that the latter is a bug, and easily fixed. I don't see much justification not to copy that value over.

The fact that the Texture object gets copied.

[abelnation]

FYI, i've updated the example page link:
http://groovemechanic.net/three.js.release/examples/webgl_lights_rectarealight.html

I plan to open a PR soon to start moving towards a final submission for this feature.

[mrdoob]

Niiice!

[MasterJames]

Thankfully I now get an error OES_texture_float_linear not supported on mobile.

[abelnation]

@MasterJames If this work gets merged, we'd probably explore using half-float textures to enable better mobile compatibility.

[MasterJames]

Okay cool I just thought to mention it so it's not over looked or too late to resolve because it's such a great feature.

[mrdoob]

Nexus 5/5x support OES_texture_float. What phone do you have?

[bhouston]

The BlinnPhong shininess exponent in the example seems to have the wrong range -- it should range between 0 and 200 or so. The upper range the example seems to be in the 10,000s - thus everything is extremely shiny.

[MasterJames]

I believe it's a couple years old now
Galaxy Note 3
http://m.gsmarena.com/samsung_galaxy_note_3-5665.php
I guess I should upgrade next year.
Still I want to stress the idea of having some kind of optimization checker that can change map resolution, or shadow methods, or lighting, or aliasing etc. In some iser adjustable default order to maintain fps or quality which could also have a way to check for supported features and fall back to for example 4 corner lights in the case of an area light or just one as an example related to this thread.
Anyway one of my PRs for updating my examples for Spot Lights was I guess too complicated for an "Example" but it was an attempted to manage the shadow map resolution that way.
The idea of error detections generating alternatives was brought up then I recall.
I believe so strongly in THREEjs as the way forward if not even the replacement for DOM that I'm only slightly worried we need to also comply with the JS mandate of one code base that runs everywhere. I'm advocating we stay vigilant in these regards and ya maybe have a special process that monitors performance and adjust for optimization. With a more realtime depth ability might be wise/needed. For example as you get closer to a shadow edge it would up the res. and when further away lower it of course too. It's kind of a memory-wise optimizer too.
Anyway little independent parts per object would be the best to start if not objectif this management process, so I had submitted a PR to have a global frame rate counter that wasn't dependent on the old stats system. Then each object could check performance and adjust itself as needed (a camera distance fov hit detect routine would also seem logical).

[abelnation]

@bhouston I currently have the upper maximum for shininess set so high so you can get the reflection to be much sharper than with a value of 200. When this is adapted for the physical lighting model (where roughness is used instead of shininess), it will be much easier to provide a simple range of values to go from completely diffuse to completely sharp specular reflections.

It is, of course, easy to change the range of the slider in the example :P

[abelnation]

@bhouston @MasterJames

I've posted a version where the BRDF computed values are converted to HalfFloat. It does not appear to work correctly on a brand new iPhone SE, despite the half_float extension being supported. The extensions are "supported" according to the device, but the specular component renders wrong on mobile :(

on my new iPhone SE, however, it seems to work correctly with the full float implementation

I've tried uses all NearestNeighbor interpolation (i.e. getting rid of linear interp) but to no avail.

http://groovemechanic.net/three.js.halffloat/examples/webgl_lights_rectarealight.html

It does appear to be functioning correctly on desktop (half float on left, float 32 on right).

[MasterJames]

It reports the same error? On this slowly aging phone. Samsung Galaxy Note 3
Finally looked online. Thanks for your awesome contributions past? Present & Future.

Approaching 10000 issues?! Gonna need a Celebration on that day! I must have "watched" a 1000 until I had to redirect my attention. Hope to be back on it some day soon (Maybe I'll have a new phone by then LOL).
I Love THREE js !!! You guys are totally rocking it!

[abelnation]

On iOS, i see the following browser error with the half float implementation:

WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL

[abelnation]

@MasterJames try the half float link one more time. i removed the check for full float support, and only show alerts when the half float extension is missing now

[bhouston]

@abelnation, there are float and half are supported on different phones, you need to basically use one or the other based on availability -- one or the other will not get you full coverage, so you need to do both if you want good coverage. Also remember there is a separate float/half extension to determine whether you can linearly interpolate float/half (one extension for each format.):

OES_texture_float
OES_texture_float_linear

And:

OES_texture_half_float
OES_texture_half_float_linear

The most compatible method would be to use int32 encoded as RGBA that you scale correctly in the pixel shader. It may be best to query adjacent samples manually and then do the linear interpolation manually to the values you need as encoded RGBA doesn't interpolate properly.

[bhouston]

@abelnation You can use http://webglstats.com/ to figure out the coverage of the various extensions. It looks like iOS now has great support for these extension but 90% of Andriod devices lack the OES_texture_float_linear and about 35% of Android devices lack the OES_texture_half_float_linear extension. So I guess the main issue is Andriod devices these days.

[MasterJames]

That's got it awesome!
Just a minor aliasing issue on the fringe now.

[abelnation]

@bhouston what's annoying is that ios webkit claims that the extensions are supported, and yet do not render properly :(

sounds like half float works on @MasterJames device, which is promising.

[MasterJames]

Again support for my thinking we need to have error detection with automatic fallback as well as optimization monitoring.
Further I recommend that maybe an API method like optimizer(stats, system, camera, userPref = "fps") that is part of the render loop every second maybe (and deleted from array if not implimented) and so if optionally implimented gives the individual entities a chance to lower aliasing, shadow map resolution, etc. as well as being able to look at supported features or maybe an error being broadcast. Anyway something to think about how to make it properly integrated so everybody's happy both devs and users. Thanks for listening and your considerations.

[bhouston]

@MasterJames, I do think that adaptive quality makes sense. I've done some experiments in that area, and Clara.io's had adaptive quality for more than a year, but it is probably best for a separate github issue.

[sam-g-steel]

Wow, I really like this feature!!!
It looks like dev on this feature...

I'd be glad to help.

Is the only thing holding this feature back the half pixel issue?
I ran the demo http://groovemechanic.net/three.js.release/examples/#webgl_lights_rectarealight on my iPhone 6s... Its working fine for me.

It also works on my HTC 10, HTC M9, Nexus 9, and Moto X Pure 2015!

@mrdoob, @abelnation if I can help Id be very glad to volunteer!

[abelnation]

@sam-g-steel Only real thing holding the PR up is a proper review by @WestLangley who wants to verify the correctness of the implementation.