8234920: Add SpotLight to the selection of 3D light types

[nlisker]

Added a SpotLight only to the D3D pipeline currently.

API discussion points

• Added SpotLight as a subclass of LightBase. However, it could also be a subclass of PointLight as it's a point light with direction and extra factors. I saw that scenario.effect.light.SpotLight extends its respective PointLight, but it's not a perfect analogy. In the end, I think it's a questions of whether PointLight will be expanded in a way which doesn't not suit SpotLight, and I tend to think that the answer is no.

• The inner and outer angles are the "diameter angles" as shown here. I, personally, find it more intuitive that these are the "radius angles", so half these angles, as used in the spotlight factor formula. Do you think I can change this or do you prefer the current definition of the angles?

• The current implementation uses an ad-hoc direction property (using a Point3D). It crossed my mind that we could use the rotation transforms of the node to control the direction instead, just like we use the translation/layout of the node to get the position (there is an internal Affine3D transform for lights, not sure why AmbientLight needs it). Wouldn't that make more sense? When I rotate the light I would expect to see a change in direction.

Implementation discussion points

• I've gotten advice from a graphics engineer to treat point lights as spot lights with a 360 degrees coverage, which simplifies a few places. We can still try to optimize for a point light by looking at the light parameters: falloff = 0 and outerAngle = 180. These possible optimization exist in ES2PhongShader.java and D3DMeshView.cc, and in the pixel/fragment shaders in the form of 3 different ways to compute the spotlight factor (the computeLightN methods). We need to check which of these give the best results.

Performance

Testing 3 point lights and comparing this branch with master using a 1000 division sphere, 200 meshes, and 5000 meshes.
Using an AMD RX 470 4GB GPU.

In this branch, there is a possible CPU optimization for checking the light type and using precalculated values (in D3DMeshView.cc for d3d and ES2PhongShader.java for opengl). On the GPU, I tried 3 ways of computing the spotlight factor contributions (computeSpotlightFactor, computeSpotlightFactor2 and computeSpotlightFactor3) trying out different branching and shortcuts.

Results

The CPU "optimizations" made no difference, which is understandable considering it will not be the bottleneck. We can remove these if we want to simplify, though maybe if we allow a large number of lights it could make a difference (I doubt it). I don't have a strong preference either way.

The sphere 1000 tests always gave max fps (120 on Win and 60 on Ubuntu).

Win 10
Compared with the master branch, this patch shows 5-10 fps drop in the mesh 200 test and ~5 in the mesh 5000 test. I repeated the tests on several occasions and got different results in terms of absolute numbers, but the relative performance difference remained more or less the same. Out of the 3 computeSpotlightFactor methods, computeSpotlightFactor3, which has no "optimizations", gives slightly better performance.

Ubuntu 18
The mesh 200 test always gave 60 fps because it is locked to this fps, so we can't measure the real GPU performance change.
The mesh 5000 test shows 2-6 fps drop from master, with computeSpotlightFactor > computeSpotlightFactor2 > computeSpotlightFactor3 at in terms of performance (~2 fps difference each).

Conclusion: we can expect a 5 fps drop more or less with 3 point lights. computeSpotlightFactor3 on d3d and computeSpotlightFactor on opengl gave the best performances.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Integration blocker

 ⚠️ The change requires a CSR request to be approved.

Issue

• JDK-8234920: Add SpotLight to the selection of 3D light types

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jfx pull/334/head:pull/334
$ git checkout pull/334

Update a local copy of the PR:
$ git checkout pull/334
$ git pull https://git.openjdk.java.net/jfx pull/334/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 334

View PR using the GUI difftool:
$ git pr show -t 334

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jfx/pull/334.diff

[bridgekeeper]

👋 Welcome back nlisker! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[nlisker]

/csr
/reviewers 2

[openjdk]

@nlisker has indicated that a compatibility and specification (CSR) request is needed for this pull request.
@nlisker please create a CSR request and add link to it in JDK-8234920. This pull request cannot be integrated until the CSR request is approved.

[openjdk]

@nlisker
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

[nlisker]

I suggest we start with looking at the API and the subclass question. This will unblock the CSR process.

[kevinrushforth]

My preference would be for SpotLight to subclass PointLight. I also somewhat prefer the 1/2 angle (radius) rather than diameter to specify the spread angles (inner and outer).

[kevinrushforth]

Not that it's all that relevant, but I will note that the (very old) Java 3D API also had SpotLight as a subclass of PointLight.

[nlisker]

Is the old implementation worth looking at, or is is completely different?

I updated the API after subclassing PointLight. After we settle on it I will submit a CSR and we can move on to the implementation.

[nlisker]

Another API point is how to implement the direction - a Point3D or 3 doubles. We've had this discussion before when talking about transforms, and didn't reach a conclusion.

[kevinrushforth]

Is the old implementation worth looking at, or is is completely different?

No, the Java 3D implementation was done using fixed-function pipeline (not shaders), so not really a good starting point.

Another API point is how to implement the direction - a Point3D or 3 doubles.

Yes, we need to sort this one out for SpotLight. This is a direction vector similar to the rotation axis in Rotate, so using Point3D for this property seems most consistent.

[nlisker]

This is a direction vector similar to the rotation axis in Rotate, so using Point3D for this property seems most consistent.

This is the current implementation. The only downside is that it's more difficult to use bindings with.

[kevinrushforth]

I think the API looks good with a few comments about valid ranges.

[openjdk]

@nlisker this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout 8234920_Add_SpotLight_to_the_selection_of_3D_light_types
git fetch https://git.openjdk.java.net/jfx master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

[nlisker]

There is an outstanding API question regarding the direction vector: Should the transforms of the SpotLight node to scene coordinates affect the direction vector (e.g., such that a rotation would rotate the direction the spotlight is pointing)? If so, do we even need the direction vector or could we define the direction as (0,0,1) in the local coordinates of the SpotLight, and tell applications to apply the appropriate rotation to define the direction.

I think for consistency, the answer is "Yes" to the first question (that's how the position works and it would be odd for the transform to affect the position and not the direction). I'm less certain about the answer to the second question. Without utility functions like lookAt to help compute the appropriate transform, it seems important to be able to specify the direction as an explicit parameter. And even if we had utility functions, an app might want the control (although you might argue it would be unneeded). My instinct is to keep it as defined. If we go with this, the only change that is needed is a note that the transforms applied to the SpotLight node affect it's direction as well as its position.

I would like to allow a developer to achieve a functionality like is shown in https://www.youtube.com/watch?v=CFgwZX5dkcM at 9:50. The rotations are intuitive there. If we allow both rotation transforms and a direction, wouldn't that cause the direction to be unintuitive? Or do you mean that the direction is always the look-at regardless of rotation transforms and if it's null then the rotations take over?

On Mac I no longer get a GLSL shader error at runtime, but spotlights aren't working correctly, either.

I don't see this on Linux and I don't have a Mac. Can you try on Linux and see what you get? If Linux works, I'm afraid I would not be able to debug the Mac.

[nlisker]

I modified the existing attenuation test to use a SpotLight, which is more general, and separated the tests for the various contributions of the light. I renamed the class, so the previous version is removed, but its tests are the same in the new class.

[kevinrushforth]

From an API coverage point of view, it seems better to leave the existing PointLight tests and add new ones for SpotLight.

[kevinrushforth]

I would like to allow a developer to achieve a functionality like is shown in https://www.youtube.com/watch?v=CFgwZX5dkcM at 9:50. The rotations are intuitive there. If we allow both rotation transforms and a direction, wouldn't that cause the direction to be unintuitive? Or do you mean that the direction is always the look-at regardless of rotation transforms and if it's null then the rotations take over?

I the rotation needs to apply to whatever the direction vector is: whether that's implicitly (0,0,1) if we don't provide a separate direction vector or whether it's the user provided direction vector, it would be odd to ignore transforms if you set the vector.

If we don't provide a vector, we could always add it later if people ask for it, so maybe it is better to leave it off if we aren't sure.

The question then becomes how easy is it to modify the direction vector using the node transforms?

On Mac I no longer get a GLSL shader error at runtime, but spotlights aren't working correctly, either.

I don't see this on Linux and I don't have a Mac. Can you try on Linux and see what you get? If Linux works, I'm afraid I would not be able to debug the Mac.

It's broken on Linux, too -- more so, since there is no lighting most of the time.

[nlisker]

I the rotation needs to apply to whatever the direction vector is: whether that's implicitly (0,0,1) if we don't provide a separate direction vector or whether it's the user provided direction vector, it would be odd to ignore transforms if you set the vector.

I see, so we compose the transforms on top of the direction vector. It's an option indeed.

If we don't provide a vector, we could always add it later if people ask for it, so maybe it is better to leave it off if we aren't sure.

The question then becomes how easy is it to modify the direction vector using the node transforms?

Yes, the ease of use is the issue I brought up initially with this approach. The rotation transforms are non-commutable, so it's going to be unintuitive for complex operations. A look-at method could be very useful not only in this case, but for any node in a 3D scene (or even 2D). Do you think that this is a worthy addition?

[nlisker]

I think I corrected the GL pipeline issue. The dot product should have used the negative light direction. It worked for me because I was testing in the -1 z direction instead of the +1. I still don't know why Mac and Linux would have given different results. This is what I get after the commit:

[nlisker]

If you want to test aiming the light with transforms instead of a direction vector, I modified the NGSpotLight and AttenLightingSample files to use the rotation transforms.

transforms.zip

[kevinrushforth]

The problem with the hole in the spotlight on Mac is fixed. The remaining problems seem to be related to the way range is treated. Maybe there is a problem with a scale somewhere? This is on a retina display with 2x scaling. Here are two sample pictures that illustrate what I mean. The first is on Windows, and the second is on Mac.

Also, I still see some unrelated problems on Linux, but since it is a VirtualBox guest running on my Windows host, I'm not sure how valid a test it is. I have to set -Dprism.forceGPU=true to get any 3D on that machine. @arapte was going to try it on a different system (also a VM image, but different system and graphics card). I'm less worried about this one as long as physical Linux box that actually reports 3D support works correctly.

So I think we're down to the range and/or scaling being wrong on Mac. Possibly in retina mode only.

As for setting the spot light direction using rotation, I think there are three API choices, and I don't like the third since it is inconsistent, so really two choices.

1. No direction property in the SpotLight API. A vector of (0,0,1) is transformed from the local coordinate space of the light into world coordinates and used as the direction.
2. Add a direction property to the SpotLight API, with a default of (0,0,1). That direction vector is transformed from the local coordinate space of the light into world coordinates and used as the direction.
3. Add a direction property to the SpotLight API, with a default of (0,0,1). That vector is use as the direction without regard to any transforms.

Option 2 is the most flexible. I think option 1 is fine, since we can always compatibly add a direction vector in the future (giving us option 2 later). As I mentioned, option 3 is inconsistent, so let's not do that one.

[nlisker]

I will check the range issue on (non-VM) Linux. I don't have a Mac, so not sure what to do if Linux checks out.

As for setting the spot light direction using rotation, I think there are three API choices, and I don't like the third since it is inconsistent, so really two choices.

The current patch uses approach 3, and I posted a couple of replacement files in the comment above that use approach 1. If you and Ambarish can compare these, we will have a better idea on what to do. I also think that either 1 or 2 are the best choices, so if you find approach 1 too confusing or 3 very convenient, I will merge them into approach 2.

[kevinrushforth]

The current patch uses approach 3

Right.

and I posted a couple of replacement files in the comment above that use approach 1.

I ran it, but didn't spend any time looking at how convenient it would be from an API point of view.

If you and Ambarish can compare these, we will have a better idea on what to do. I also think that either 1 or 2 are the best choices, so if you find approach 1 too confusing or 3 very convenient, I will merge them into approach 2.

I'm sort of leaning towards 2 (as you could probably guess from my comments above). I'd like to hear from Ambarish as well.

[nlisker]

I ran it, but didn't spend any time looking at how convenient it would be from an API point of view.

I meant convenient for the user. My main concern is that just using rotation transforms will be frustrating because they don't always behave very intuitively. In any case, since we are going to have rotations work anyway, I'll implement option 2 with both rotations and direction, and we can check if the direction helps (leaving the direction at its default is effectively option 1).

[nlisker]

I update the patch to include rotation transforms applying on the direction vector.

When testing, I suggest using the "Boxes" model. You will need to rotate (right/middle mouse buttons) to see model properly as the back wall obscures the camera's initial position.

[arapte]

Provided few comments on documentation. Have to review and test the code.

[arapte]

->
A {@code SpotLight} is a {@code PointLight} that radiates a cone of light in a specific direction.
< p >
The direction is defined by the {@link #directionProperty() direction} property.
The light cone is defined by 3 properties: an {@link #innerAngleProperty() inner angle}, an {@link #outerAngleProperty() outer angle}, and a {@link #falloffProperty() falloff} property.

[arapte]

Also Will it be illustrative to mention these three properties using li tags with a statement that briefs the use of property.
Something like,

• {@link #innerAngleProperty() inner angle}: Angle of the inner cone where light intensity is always maximum.
• {@link #outerAngleProperty() outer angle}: Angle of the outer cone where light intensity attenuates based on fallof.
• {@link #falloffProperty() falloff}: The attenuation factor that controls the attenuation of light intensity from edge of inner cone to the edge of outer cone.

[arapte]

Should we have cone depicted in the diagram ?

[arapte]

->
For a point whose direction vector from the source of light forms an angle {@code theta} with the direction of light,
if {@code theta < innerAngle} then that point receives maximum illumination.
if {@code theta > outerAngle} then that point receives no illumination. and
if {@code innerAngle <= a <= outerAngle} then the illumination received by the point is determined by formula

[arapte]

Minor: missing of: // To initialize the class helper at the beginning of each constructor of this class

[arapte]

-> The direction vector of the spotlight. The vector need not be normalized.

[arapte]

We should use Point or vertex instead of word surface. Using Point would be consistent with description of class.

[nlisker]

I'll fix the documentation once we establish how the light is rotated. I will need to add notes about the rotation transforms too.

[nlisker]

@kevinrushforth I have just tested on Ubuntu 20 and I don't see any issue (at least not something that's different than Windows). If it's an issue with the range property, does it happen with point lights too? Does it happen before this patch (but with the attenuation patch)? Also, does it happen when there is only 1 light turned on or also more (there are different files for the number of lights)?

If you can give the exact values for the properties that cause this I will be able to try a direct reproduction. I tested the animating sphere. I suggest to use the Boxes models since they are not animated and the range is constant.

[kevinrushforth]

I'll try it on macOS with a non-retina display and see if that makes a difference (if so, it could point to a transform issue). I'll also try with the Boxes as you suggest.

[kevinrushforth]

Yep, there's definitely a scale problem going on. It looks much different if I run it on a non-Retina display. And knowing that, I was able to reproduce it on Windows as well by comparing what it looks like with -Dglass.win.uiScale=1 and -Dglass.win.uiScale=2.

[nlisker]

I asked someone with a Mac Book Pro with a retina screen to run JavaFX 16, and I saw an issue with lighting positioning: the source of the light was not the position of the PointLight node. I couldn't debug it, but it seems that this issue is not related to this patch, and possibly not even to the range property specifically.

[nlisker]

Can we continue with the review? We need to decide about the rotation behavior and the performance options need to be tested.

[kevinrushforth]

I started to look at this today, and will continue next week. There is also the outstanding issue with a retina display (scale=2) on macOS.

[nlisker]

There is also the outstanding issue with a retina display (scale=2) on macOS.

Yes, it has to be looked at, but I think it is unrelated to this particular patch. If this is the case, we can file a different issue and continue with this one.

[kevinrushforth]

You may be right about the scale problem being preexisting. I'll double check when I test on Mac next week (both Retina and external screen).