Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Projectors to support textured lighting #13057

Open
wants to merge 6 commits into
base: dev
from

Conversation

Projects
None yet
@usefulthink
Copy link
Contributor

commented Jan 7, 2018

Adds a new light-type ProjectorLight, a light-source with a textured rectangular beam with configurable aspect and rotation.

Example: https://cdn.rawgit.com/usefulthink/three.js/projector-lights-preview/examples/webgl_lights_projectorlight.html

Usage:

var projector = new THREE.ProjectorLight(0xffffff, 1);
projector.fov = 20; // vertical FOV in degrees
projector.aspect = 16 / 9; // texture aspect-ratio
projector.map = new THREE.TextureLoader().load(textureUrl);

scene.add(projector);

Note: this PR is not yet fully completed and there are some minor parts I am not really happy with. I am posting it mostly to get some early feedback on general interest from other developers and feedback on implementation details.

implement ProjectorLights to support textured lighting
Adds a new light-type `ProjectorLight`.
ProjectorLights are lights with an rectangular beam with arbitrary
aspect and rotation that can be textured using a regular texture.

Example:

    var projector = new THREE.ProjectorLight(0xffffff, 1);
    projector.fov = 20; // vertical FOV in degrees
    projector.aspect = 16 / 9; // texture aspect-ratio
    projector.map = new THREE.TextureLoader().load(textureUrl);

    scene.add(projector);
@takahirox

This comment has been minimized.

Copy link
Collaborator

commented Jan 7, 2018

Looks interesting...

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2018

I am posting it mostly to get some early feedback on general interest from other developers and feedback on implementation details.

An altered SpotLight is always useful.
👍

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 7, 2018

This is great!

screen shot

Bat-signal implemented with a grayscale RGB texture.

One suggestion would be to attenuate the intensity with the alpha channel as well.

vec4 texture = texture2D( projectorTexture, projectorUv );

directLight.color = projectorLight.color;
directLight.color *= texture.rgb;
directLight.color *= texture.a; // attenuate

I also think support for texture offset/repeat/center/rotation would be a worthwhile addition.

@tentone

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2018

@usefulthink Thanks a lot! Nice!

I would aim into merging it as it is, and improve it on new pull requests!

usefulthink added some commits Jan 7, 2018

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 7, 2018

@WestLangley thinking out loud here, but shouldn't it be possible with relatively few changes to support both perspective and orthographic modes (i.e. ProjectiveSpotLight and ProjectiveDirectionalLight) with the same basic light-type or am I missing something here?

Good point with the attenuation from alpha-channel, added that.
I'm not yet familiar with how the texture-parameters (offset/repeat/center/rotation) are handled, will dig into that and see if it can be added.

@looeee

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

This is a great addition.

Would it be better to add a .projectiveMap parameter to existing SpotLight and DirectionalLight, rather than creating a new light type though?

We could add a coneShape (and aspect ) settings to SpotLight too, to switch between square or circular cones.

@takahirox

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

I prefer adding projective map property rather than adding a new light type if it works fine.

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

I would keep these new lights as separate classes and separate from each other at first, and merge the feature set into existing lights later only if it makes sense.

@looeee

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

@WestLangley that sounds like a way to grow THREE.Legacy.js even faster. Is there a reason why it doesn't make sense now?

We should agree on a final API before merging new features, rather than adding something that we intend to change later.

@Mugen87

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

I tend to vote for @WestLangley's suggestion.

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2018

also added support for uvTransform now so all common texture-features should now be supported.

@looeee @takahirox I first tried implementing it as a parameter for the spotlight, but it turns out that it effectively results in mostly seperate code-paths and lots of complications within the shaders and in WebGLLights.js. Those could be combined to some degree (if we see a spotlight as a special case of a projective light), but that would slightly slow down the codepath for regular spotlights.

@takahirox

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

"if it works fine" I mentioned meant "I prefer map property approach if we can easily do", so that's ok if it's difficult. (I haven't read through the implementation yet).

BTW, curious to know if similar project-light type exists in other engines, for example unity.

@Mugen87

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

BTW, curious to know if similar project-light type exists in other engines, for example unity.

As far as i know, Unity has no separate light types for this. One concept they use is Cookies.

See https://www.youtube.com/watch?v=WERan6gjEn4

It's also possible to create similar effects with Projector, although this entity is focused on material/shadow projection.

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2018

I searched through some of the docs:

  • unreal does this using "light-functions"
  • in cryengine it's a property of the light-entities (although limited to 512x512 quadratic textures)
  • unity supports "cookies" as textures on all types of lights.

So yes, all of them seem to implement it as part of the regular lights, and I think we could probably get there at some point.

@takahirox

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

Thanks

I think we could probably get there at some point.

👍

@Mugen87

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

If we would enhance the basic light types, i would vote for the Cookies approach of Unity. This would only work grayscale textures but you could achieve the same effect like in @WestLangley's demo (with a SpotLight and a respective map).

@looeee

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

@Mugen87 is that for performance reasons? Seems a shame to limit this to B&W if we can do full color and videos.

@Mugen87

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

From a conceptual point of view, it might be a good idea to treat such maps as pure shaped masks, like mentioned in the Unity docs.

For shadow and material/texture projections (which can also be used to fake light effects) Unity uses a more general Projector class. I've worked a little bit with Unity in the past and i liked this separation. So it's just a personal preference 😊 .

More information about Projectors: https://www.youtube.com/watch?v=44Nad3QwuoA&

@looeee

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

I was thinking of it more in terms of modelling of a physical process - that is, a light being partially or fully occluded while shining past / through objects in front of it. In the case of a real world projector, the occluding object is a semi-transparent LCD screen. Other examples are stained glass or a narrow window.

Since unity cookies allow grayscale, they are not actually just cutouts and are modelling a limited version of this physical process.

On the other hand, I presume that the reason cookies and projectors are separate in Unity is that cookies just multiply the light's intensity by the gray level of the cookie image, which is probably a cheap operation.
Projectors have to adjust the color and the intensity which may be more costly, so it would make more sense to keep them separate if this is the case.

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2018

Would it be better to add a .projectiveMap parameter to existing SpotLight

This would be the best approach in my opinion.

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

I like the Projector nomenclature as a class name.

It could support both perspective an orthographic frustums, and a color-map, cookie, or video.

It would serve as an additional light source and cast shadows.

A shortcoming may be the specular highlights generated by the projector. They may not be correct. RectAreaLight has proper specular highlights.

As I said, I would keep it as a separate class and leave existing lights alone until this is sorted out.

@Mugen87

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

Just a little addition on Cookies. They are actually alpha maps but Unity offers an option to convert the luminosity expressed in a grayscale texture to alpha. If we ever add Cookies to three.js we should do the same 😁 .

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 8, 2018

Unity offers an option to convert the luminosity expressed in a grayscale texture to alpha.

@Mugen87 That is not required. The code I posted above is sufficient to handle all cases.

All we need is a Projector.map property which can be an RGB or RBGA texture, or a video texture.

@usefulthink usefulthink changed the title Implement ProjectorLights to support textured lighting Implement Projectors to support textured lighting Jan 8, 2018

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 8, 2018

Good points here, I did go with @WestLangley and removed the Light suffix from the Projector, I like that a lot more.

Regarding the specular highlights, I don't think I can solve that with my current knowledge of how the BSDFs work. I'm updating the IncidentLight with the color seen at the fragment, and to me the highlights look quite plausible, what am I missing?


if ( shadow.isProjectorShadow ) {

shadowCamera.rotateZ( light.rotation.z );

This comment has been minimized.

Copy link
@WestLangley

WestLangley Jan 11, 2018

Collaborator

I'm not sure this property adds much. Consider removing it?

This comment has been minimized.

Copy link
@usefulthink

usefulthink Jan 11, 2018

Author Contributor

What I want to achieve is full 3DOF rotation of the projector. The first two are already handled by the lookAt(target) I copied from the other light-types, so there needs to be a way to rotate the projector relative to it's up-direction. If this would be removed, the only option to do that would be to modify the up-direction which feels a bit weird.

One refactoring I would love to do is to make the target optional for all light-types: If no target is specified, the default Object3D.rotation property would be used instead. That would allow me to implement this rotation in client-code if required.

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 12, 2018

Sorry to ruin the party but Projector is taken 😅
https://github.com/mrdoob/three.js/blob/dev/examples/js/renderers/Projector.js#L99

TextureProjector maybe?

I favour adding texture or map as property to lights though, otherwise it seems like it would end up in a lot of duplicate code. And I favour of keeping it RGBA, basically using the texture as a multiplier of the light.

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 12, 2018

I favour adding texture or map as property to lights though

The light in this PR has a rectangular frustum. It is a model for a studio light with so-called "barn doors" that can be used to control the shape of the light.

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

With SpotLight, the two frustums are decoupled; the spotlight can have a wide angle, and the shadow frustum can be tight.

If you just want to add a map property to SpotLight, then we need to do some thinking...

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 12, 2018

And I favour of keeping it RGBA, basically using the texture as a multiplier of the light.

I think that is what we agreed on above.

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 12, 2018

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

We could also add add a RectLight?

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 13, 2018

We could also add add a RectLight?

I'm not exactly sure what you are asking, but if it pertains to adding a map property to RectAreaLight, the answer is basically no: the light would not 'project' the map.

Real projectors use lenses to emit nearly-parallel beams of light. Our spotlights and directional lights are modeled as such emitters.

RectAreaLight models something akin to a computer monitor. Light is emitted in every hemispherical direction, and consequently, it does not project images well.

@looeee

This comment has been minimized.

Copy link
Collaborator

commented Jan 13, 2018

What about adding SpotRectLight?

Or adding a parameter to SpotLight to control the frustum shape?

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 13, 2018

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

As I said before,this "projector" is an altered SpotLight and in my opinion the best approach would be to add a few more parameters for this light.

  • one for frustum (conical or rectangular)
  • one for projected map (none or texture)
    etc
@haroldiedema

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2018

RectAreaLight models something akin to a computer monitor. Light is emitted in every hemispherical direction, and consequently, it does not project images well.

I have to disagree here. It completely depends on what kind of effect you're trying to achieve of course, but imagine having a "roster" in front of your lights, it would project the texture precisely as you'd expect without the need of using "real" shadows.

I vote for adding the extra properties to the existing lights, as @RemusMar said.

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 15, 2018

@haroldiedema An image added to a RectAreaLight will reflect specularly in other objects, but the light will not project the image any more than a computer monitor projects its image.

Sorry, I do not understand what you are trying to say. What do you mean by "roster", and what does it have to do with shadows?

@haroldiedema

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2018

@haroldiedema An image added to a RectAreaLight will reflect specularly in other objects, but the light will not project the image any more than a computer monitor projects its image.

Oh my apologies, then I misinterpret the implementation of RectAreaLight..

Sorry, I do not understand what you are trying to say. What do you mean by "roster", and what does it have to do with shadows?

I mean that with this, you can effectively fake shadows.
Like this image

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 15, 2018

I mean that with this, you can effectively fake shadows.

Right. And that is a compelling reason for this PR. Projector can do that. RectAreaLight cannot -- even if texture support were added.

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 19, 2018

We could also add add a RectLight?

I'm not exactly sure what you are asking, but if it pertains to adding a map property to RectAreaLight, the answer is basically no: the light would not 'project' the map.

No no, I'm proposing creating a new type of light (RectLight) which would basically be a SpotLight but without the circle code.

I think we're all suggesting the same.

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 19, 2018

I'm proposing creating a new type of light (RectLight) which would basically be a SpotLight but without the circle code.

Why Rect if this is a Spot?
The frustum can be: triangular, rectangular, ..., hexagonal, ..., circular (cone).

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 19, 2018

spot
noun
1. a small round or roundish mark

🤔

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 19, 2018

@mrdoob I would like to focus the name more on the texturing than on the beam-shape, you can achieve any shape you want with it... What do think about ProjectorLight, TexturedLight or something like that?

Also, would you prefer a fully integrated approach (i.e. additional properties on SpotLight) or this "standalone" solution?

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 19, 2018

spot
noun

  1. a small round or roundish mark

SpotLight has a conical frustum which will expand up to 180 degrees, but a rectangular shadow frustum.

Or maybe a "roundish" shadow frustum for a better SPOT.
ha ha ha

@mrdoob

This comment has been minimized.

Copy link
Owner

commented Jan 19, 2018

If we follow 3D Studio implementation, it would be a matter of adding shape (Circle, Rectangle) and aspect to SpotLight.

guid-75442a44-9424-4a77-805a-7dd95b5e1792

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 19, 2018

So in that case, I would propose implementing it like this (hopefully I can tackle that over the weekend):

  • spotLight.beamShape (constants Circle or Rectangle): controls the shape of the beam

  • spotLight.aspect (number): width/height ratio of the beam for rectangular beams (spotLight.angle is interpreted as vertical FOV if aspect is specified)

  • spotLight.map (Texture): projected texture. Should work with both beam-shapes

    • rectangular: texture completely fills the beam-region
    • round: texture is clipped by the beam
  • handle rotation around the look-axis (see #13057 (comment)). I'd implement the target/rotation refactoring mentioned in that comment (for all light-types: make target-property optional, if no target is specified, the default light.rotation would be used instead).

I dind't think too much about other light-types yet (and will probably not implement them right away). PointLight should be pretty simple: we would just use the map-property (in this case a CubeTexture), all others don't really make sense. Texture-lookups can probably reuse the logic from shadowmap-lookups.

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 19, 2018

spotLight.beamShape, spotLight.aspect

I would highly recommend a simpler solution if you want to add a map property to Spotlight: set the rectangular frustum to be just large enough to enclose the existing conical frustum. That means it will be square, and defined by Spotlight.angle. That way it will adapt to the angle.

You only need to add one property, map, to SpotLight.

Also, it is very important that the shadow frustum be separate from this frustum so we can still support tight, quality shadows.

for all light-types: make target-property optional,

If you want to propose that, please, please do it in a separate PR.

handle rotation around the look-axis

This seams to be an extra feature you are pushing for. I would omit it. Just add the map. Texture maps can be rotated now, which achieves the same effect.

PointLight should be pretty simple

I'm not sure about PointLight, but support for DirectionalLight.map is definitely a good idea, and should have priority. Which frustum to use in the case of directional light is to-be-determined.

@RemusMar

This comment has been minimized.

Copy link
Contributor

commented Jan 19, 2018

set the rectangular frustum to be just large enough to enclose the existing conical frustum.
That means it will be square, and defined by Spotlight.angle

Square is also a better option for textures.
👍

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 19, 2018

@WestLangley I think we are thinking about different usage-scenarios.

What would you think about adding a map-property to the SpotLight (working in a simplified way as you described) and keeping this class (renamed to whatever we decide on) for the full-fledged projector use-case?

That means it will be square, and defined by Spotlight.angle. That way it will adapt to the angle.
You only need to add one property, map, to SpotLight.

A square frustum with the normal round beam would be the default, but I wouldn't want to stop there as it doesn't really cover what I am about to do: I want to simulate actual projectors, so I need for instance a beam with 16:9 aspect-ratio using a texture from a 16:9 video-source. I don't see how that could be achieved with a square-beam without having to resample the video to a size that fits inside the spotlights beam.

Also note that supporting the aspect/shape cases won't necessarily need much extra-work in the shaders and no extra uniforms (well, at least if it's implemented the way I did it, there may be more efficient ways).

Also, it is very important that the shadow frustum be separate from this frustum so we can still support tight, quality shadows.

I will consider that, but I would still try to provide a default that covers the whole light-volume of the light, ideally nothing more.

If you want to propose that, please, please do it in a separate PR.

Will do.

This seams to be an extra feature you are pushing for.
I would omit it. Just add the map. Texture maps can be rotated now, which achieves the same effect.

😄 it is, mostly because I think it wouldn't be complete if we left a useful degree-of-freedom inaccessible to the user. Not sure about that "same effect" – especially the shadow-camera will not know that the texture was rotated. What I don't like about the texture-rotation is that it doesn't work well with a shadow-camera that is synchronized with the light-volume (which should be the default behaviour).

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Jan 20, 2018

I want to simulate actual projectors, so I need for instance a beam with 16:9 aspect-ratio

A new light projector class is fine with me (leaving the existing lights as-is), but the others in this thread prefer you just add a map property to Spotlight and DirectionalLight. That means keeping it simple. It is also the best way to get your PR merged.

In real life, studio lights have "barn doors", as I mentioned. You can model those with "cookies", so the conical frustum is masked, and renders as a rectangular one.

Your other option is to continue to argue for the separate ProjectorLight.

@usefulthink

This comment has been minimized.

Copy link
Contributor Author

commented Jan 20, 2018

That means keeping it simple. It is also the best way to get your PR merged.

Jip. Will give that a try then. I'd still hope to keep the full projector-implementation around :)

The only question I have regarding the spotlight.map-version is how to handle the texture-uniforms properly. If I declare them as

uniform sampler2D spotLightTextures[NUM_SPOT_LIGHTS];

but only assign a texture to one of the indices, would that waste any gpu-resources for the lights without textures?

@mbredif

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2018

#14621 has a proposal to add the map property to spotlights.
It also refrains from using a texture slot if the map property is undefined. (addressing the concern #13057 (comment)).

@mrdoob , @WestLangley, which alternative would you prefer getting merged ? as a new Light type (this PR) or as a SpotLight with a map property (#14621, to be reviewed) ?

@WestLangley

This comment has been minimized.

Copy link
Collaborator

commented Aug 7, 2018

@mbredif Based on the discussion in this thread, adding .map to SpotLight appears to be the preference.

@sndrgb

This comment has been minimized.

Copy link

commented Sep 23, 2018

Any news about this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.