Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8234920: Add SpotLight to the selection of 3D light types #334

Open
wants to merge 32 commits into
base: master
Choose a base branch
from

Conversation

@nlisker
Copy link
Collaborator

@nlisker nlisker commented Oct 22, 2020

Added a SpotLight only to the D3D pipeline currently.

API discussion points

  • Added SpotLight as a subclass of LightBase. However, it could also be a subclass of PointLight as it's a point light with direction and extra factors. I saw that scenario.effect.light.SpotLight extends its respective PointLight, but it's not a perfect analogy. In the end, I think it's a questions of whether PointLight will be expanded in a way which doesn't not suit SpotLight, and I tend to think that the answer is no.

  • The inner and outer angles are the "diameter angles" as shown here. I, personally, find it more intuitive that these are the "radius angles", so half these angles, as used in the spotlight factor formula. Do you think I can change this or do you prefer the current definition of the angles?

  • The current implementation uses an ad-hoc direction property (using a Point3D). It crossed my mind that we could use the rotation transforms of the node to control the direction instead, just like we use the translation/layout of the node to get the position (there is an internal Affine3D transform for lights, not sure why AmbientLight needs it). Wouldn't that make more sense? When I rotate the light I would expect to see a change in direction.

Implementation discussion points

  • I've gotten advice from a graphics engineer to treat point lights as spot lights with a 360 degrees coverage, which simplifies a few places. We can still try to optimize for a point light by looking at the light parameters: falloff = 0 and outerAngle = 180. These possible optimization exist in ES2PhongShader.java and D3DMeshView.cc, and in the pixel/fragment shaders in the form of 3 different ways to compute the spotlight factor (the computeLightN methods). We need to check which of these give the best results.

Performance

Testing 3 point lights and comparing this branch with master using a 1000 division sphere, 200 meshes, and 5000 meshes.
Using an AMD RX 470 4GB GPU.

In this branch, there is a possible CPU optimization for checking the light type and using precalculated values (in D3DMeshView.cc for d3d and ES2PhongShader.java for opengl). On the GPU, I tried 3 ways of computing the spotlight factor contributions (computeSpotlightFactor, computeSpotlightFactor2 and computeSpotlightFactor3) trying out different branching and shortcuts.

Results

The CPU "optimizations" made no difference, which is understandable considering it will not be the bottleneck. We can remove these if we want to simplify, though maybe if we allow a large number of lights it could make a difference (I doubt it). I don't have a strong preference either way.

The sphere 1000 tests always gave max fps (120 on Win and 60 on Ubuntu).

Win 10
Compared with the master branch, this patch shows 5-10 fps drop in the mesh 200 test and ~5 in the mesh 5000 test. I repeated the tests on several occasions and got different results in terms of absolute numbers, but the relative performance difference remained more or less the same. Out of the 3 computeSpotlightFactor methods, computeSpotlightFactor3, which has no "optimizations", gives slightly better performance.

Ubuntu 18
The mesh 200 test always gave 60 fps because it is locked to this fps, so we can't measure the real GPU performance change.
The mesh 5000 test shows 2-6 fps drop from master, with computeSpotlightFactor > computeSpotlightFactor2 > computeSpotlightFactor3 at in terms of performance (~2 fps difference each).

Conclusion: we can expect a 5 fps drop more or less with 3 point lights. computeSpotlightFactor3 on d3d and computeSpotlightFactor on opengl gave the best performances.


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Integration blocker

 ⚠️ The change requires a CSR request to be approved.

Issue

  • JDK-8234920: Add SpotLight to the selection of 3D light types

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jfx pull/334/head:pull/334
$ git checkout pull/334

Update a local copy of the PR:
$ git checkout pull/334
$ git pull https://git.openjdk.java.net/jfx pull/334/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 334

View PR using the GUI difftool:
$ git pr show -t 334

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jfx/pull/334.diff

@bridgekeeper
Copy link

@bridgekeeper bridgekeeper bot commented Oct 22, 2020

👋 Welcome back nlisker! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Oct 22, 2020

/csr
/reviewers 2

@openjdk openjdk bot added the csr label Oct 22, 2020
@openjdk
Copy link

@openjdk openjdk bot commented Oct 22, 2020

@nlisker has indicated that a compatibility and specification (CSR) request is needed for this pull request.
@nlisker please create a CSR request and add link to it in JDK-8234920. This pull request cannot be integrated until the CSR request is approved.

@openjdk
Copy link

@openjdk openjdk bot commented Oct 22, 2020

@nlisker
The number of required reviews for this PR is now set to 2 (with at least 1 of role reviewers).

nlisker added 2 commits Oct 23, 2020
@kevinrushforth kevinrushforth requested review from arapte and kevinrushforth Oct 24, 2020
nlisker added 2 commits Oct 25, 2020
@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Oct 26, 2020

I suggest we start with looking at the API and the subclass question. This will unblock the CSR process.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Oct 27, 2020

My preference would be for SpotLight to subclass PointLight. I also somewhat prefer the 1/2 angle (radius) rather than diameter to specify the spread angles (inner and outer).

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Oct 27, 2020

Not that it's all that relevant, but I will note that the (very old) Java 3D API also had SpotLight as a subclass of PointLight.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Oct 30, 2020

Is the old implementation worth looking at, or is is completely different?

I updated the API after subclassing PointLight. After we settle on it I will submit a CSR and we can move on to the implementation.

nlisker added 3 commits Oct 31, 2020
@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Nov 2, 2020

Another API point is how to implement the direction - a Point3D or 3 doubles. We've had this discussion before when talking about transforms, and didn't reach a conclusion.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Nov 6, 2020

Is the old implementation worth looking at, or is is completely different?

No, the Java 3D implementation was done using fixed-function pipeline (not shaders), so not really a good starting point.

Another API point is how to implement the direction - a Point3D or 3 doubles.

Yes, we need to sort this one out for SpotLight. This is a direction vector similar to the rotation axis in Rotate, so using Point3D for this property seems most consistent.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Nov 10, 2020

This is a direction vector similar to the rotation axis in Rotate, so using Point3D for this property seems most consistent.

This is the current implementation. The only downside is that it's more difficult to use bindings with.

Copy link
Member

@kevinrushforth kevinrushforth left a comment

I think the API looks good with a few comments about valid ranges.

nlisker added 2 commits Dec 19, 2020
@openjdk
Copy link

@openjdk openjdk bot commented Dec 22, 2020

@nlisker this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout 8234920_Add_SpotLight_to_the_selection_of_3D_light_types
git fetch https://git.openjdk.java.net/jfx master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push
@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Mar 21, 2021

There is an outstanding API question regarding the direction vector: Should the transforms of the SpotLight node to scene coordinates affect the direction vector (e.g., such that a rotation would rotate the direction the spotlight is pointing)? If so, do we even need the direction vector or could we define the direction as (0,0,1) in the local coordinates of the SpotLight, and tell applications to apply the appropriate rotation to define the direction.

I think for consistency, the answer is "Yes" to the first question (that's how the position works and it would be odd for the transform to affect the position and not the direction). I'm less certain about the answer to the second question. Without utility functions like lookAt to help compute the appropriate transform, it seems important to be able to specify the direction as an explicit parameter. And even if we had utility functions, an app might want the control (although you might argue it would be unneeded). My instinct is to keep it as defined. If we go with this, the only change that is needed is a note that the transforms applied to the SpotLight node affect it's direction as well as its position.

I would like to allow a developer to achieve a functionality like is shown in https://www.youtube.com/watch?v=CFgwZX5dkcM at 9:50. The rotations are intuitive there. If we allow both rotation transforms and a direction, wouldn't that cause the direction to be unintuitive? Or do you mean that the direction is always the look-at regardless of rotation transforms and if it's null then the rotations take over?

On Mac I no longer get a GLSL shader error at runtime, but spotlights aren't working correctly, either.

I don't see this on Linux and I don't have a Mac. Can you try on Linux and see what you get? If Linux works, I'm afraid I would not be able to debug the Mac.

@openjdk openjdk bot removed the rfr label Mar 22, 2021
@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Mar 22, 2021

I modified the existing attenuation test to use a SpotLight, which is more general, and separated the tests for the various contributions of the light. I renamed the class, so the previous version is removed, but its tests are the same in the new class.

@openjdk openjdk bot added the rfr label Mar 22, 2021
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 22, 2021

From an API coverage point of view, it seems better to leave the existing PointLight tests and add new ones for SpotLight.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Mar 22, 2021

I would like to allow a developer to achieve a functionality like is shown in https://www.youtube.com/watch?v=CFgwZX5dkcM at 9:50. The rotations are intuitive there. If we allow both rotation transforms and a direction, wouldn't that cause the direction to be unintuitive? Or do you mean that the direction is always the look-at regardless of rotation transforms and if it's null then the rotations take over?

I the rotation needs to apply to whatever the direction vector is: whether that's implicitly (0,0,1) if we don't provide a separate direction vector or whether it's the user provided direction vector, it would be odd to ignore transforms if you set the vector.

If we don't provide a vector, we could always add it later if people ask for it, so maybe it is better to leave it off if we aren't sure.

The question then becomes how easy is it to modify the direction vector using the node transforms?

On Mac I no longer get a GLSL shader error at runtime, but spotlights aren't working correctly, either.

I don't see this on Linux and I don't have a Mac. Can you try on Linux and see what you get? If Linux works, I'm afraid I would not be able to debug the Mac.

It's broken on Linux, too -- more so, since there is no lighting most of the time.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Mar 25, 2021

I the rotation needs to apply to whatever the direction vector is: whether that's implicitly (0,0,1) if we don't provide a separate direction vector or whether it's the user provided direction vector, it would be odd to ignore transforms if you set the vector.

I see, so we compose the transforms on top of the direction vector. It's an option indeed.

If we don't provide a vector, we could always add it later if people ask for it, so maybe it is better to leave it off if we aren't sure.

The question then becomes how easy is it to modify the direction vector using the node transforms?

Yes, the ease of use is the issue I brought up initially with this approach. The rotation transforms are non-commutable, so it's going to be unintuitive for complex operations. A look-at method could be very useful not only in this case, but for any node in a 3D scene (or even 2D). Do you think that this is a worthy addition?

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Mar 25, 2021

I think I corrected the GL pipeline issue. The dot product should have used the negative light direction. It worked for me because I was testing in the -1 z direction instead of the +1. I still don't know why Mac and Linux would have given different results. This is what I get after the commit:

Screenshot from 2021-03-25 03-59-57

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Mar 26, 2021

If you want to test aiming the light with transforms instead of a direction vector, I modified the NGSpotLight and AttenLightingSample files to use the rotation transforms.

transforms.zip

@kevinrushforth kevinrushforth self-requested a review Apr 1, 2021
@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 9, 2021

The problem with the hole in the spotlight on Mac is fixed. The remaining problems seem to be related to the way range is treated. Maybe there is a problem with a scale somewhere? This is on a retina display with 2x scaling. Here are two sample pictures that illustrate what I mean. The first is on Windows, and the second is on Mac.

SpotLight-range-Windows
SpotLight-range-macOS

Also, I still see some unrelated problems on Linux, but since it is a VirtualBox guest running on my Windows host, I'm not sure how valid a test it is. I have to set -Dprism.forceGPU=true to get any 3D on that machine. @arapte was going to try it on a different system (also a VM image, but different system and graphics card). I'm less worried about this one as long as physical Linux box that actually reports 3D support works correctly.

So I think we're down to the range and/or scaling being wrong on Mac. Possibly in retina mode only.

As for setting the spot light direction using rotation, I think there are three API choices, and I don't like the third since it is inconsistent, so really two choices.

  1. No direction property in the SpotLight API. A vector of (0,0,1) is transformed from the local coordinate space of the light into world coordinates and used as the direction.
  2. Add a direction property to the SpotLight API, with a default of (0,0,1). That direction vector is transformed from the local coordinate space of the light into world coordinates and used as the direction.
  3. Add a direction property to the SpotLight API, with a default of (0,0,1). That vector is use as the direction without regard to any transforms.

Option 2 is the most flexible. I think option 1 is fine, since we can always compatibly add a direction vector in the future (giving us option 2 later). As I mentioned, option 3 is inconsistent, so let's not do that one.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 9, 2021

I will check the range issue on (non-VM) Linux. I don't have a Mac, so not sure what to do if Linux checks out.

As for setting the spot light direction using rotation, I think there are three API choices, and I don't like the third since it is inconsistent, so really two choices.

The current patch uses approach 3, and I posted a couple of replacement files in the comment above that use approach 1. If you and Ambarish can compare these, we will have a better idea on what to do. I also think that either 1 or 2 are the best choices, so if you find approach 1 too confusing or 3 very convenient, I will merge them into approach 2.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 9, 2021

The current patch uses approach 3

Right.

and I posted a couple of replacement files in the comment above that use approach 1.

I ran it, but didn't spend any time looking at how convenient it would be from an API point of view.

If you and Ambarish can compare these, we will have a better idea on what to do. I also think that either 1 or 2 are the best choices, so if you find approach 1 too confusing or 3 very convenient, I will merge them into approach 2.

I'm sort of leaning towards 2 (as you could probably guess from my comments above). I'd like to hear from Ambarish as well.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 9, 2021

I ran it, but didn't spend any time looking at how convenient it would be from an API point of view.

I meant convenient for the user. My main concern is that just using rotation transforms will be frustrating because they don't always behave very intuitively. In any case, since we are going to have rotations work anyway, I'll implement option 2 with both rotations and direction, and we can check if the direction helps (leaving the direction at its default is effectively option 1).

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 15, 2021

I update the patch to include rotation transforms applying on the direction vector.

When testing, I suggest using the "Boxes" model. You will need to rotate (right/middle mouse buttons) to see model properly as the back wall obscures the camera's initial position.

Copy link
Member

@arapte arapte left a comment

Provided few comments on documentation. Have to review and test the code.

* A {@code PointLight} that radiates light in a cone in a specific direction. The direction of the {@code SpotLight} is
* defined by the {@link #directionProperty() direction} property.
* <p>
* The light cone is defined by 3 factors: an {@link #innerAngleProperty() inner angle}, an {@link #outerAngleProperty()
* outer angle}, and a {@link #falloffProperty() falloff} factor. For a point whose angle to the light is {@code a}, if
Comment on lines +42 to +46

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

->
A {@code SpotLight} is a {@code PointLight} that radiates a cone of light in a specific direction.
< p >
The direction is defined by the {@link #directionProperty() direction} property.
The light cone is defined by 3 properties: an {@link #innerAngleProperty() inner angle}, an {@link #outerAngleProperty() outer angle}, and a {@link #falloffProperty() falloff} property.

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

Also Will it be illustrative to mention these three properties using li tags with a statement that briefs the use of property.
Something like,

  • {@link #innerAngleProperty() inner angle}: Angle of the inner cone where light intensity is always maximum.
  • {@link #outerAngleProperty() outer angle}: Angle of the outer cone where light intensity attenuates based on fallof.
  • {@link #falloffProperty() falloff}: The attenuation factor that controls the attenuation of light intensity from edge of inner cone to the edge of outer cone.
* behavior of the drop. The valid ranges for these properties are {@code 0 <= innerAngle <= outerAngle <= 180} and
* {@code falloff >= 0}; values outside either of these ranges can produce unexpected results.
* <p>
* <img src="doc-files/spotlight.png" alt="Image of the Spotlight">

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

Should we have cone depicted in the diagram ?

* {@code a < innerAngle} then that point receives maximum illumination, if {@code a > outerAngle} then that point
* receives no illumination, and if {@code innerAngle <= a <= outerAngle} then the illumination is determined by the
* formula
Comment on lines +47 to +49

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

->
For a point whose direction vector from the source of light forms an angle {@code theta} with the direction of light,
if {@code theta < innerAngle} then that point receives maximum illumination.
if {@code theta > outerAngle} then that point receives no illumination. and
if {@code innerAngle <= a <= outerAngle} then the illumination received by the point is determined by formula

}

{
// To initialize the class helper at the beginning each constructor of this class

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

Minor: missing of: // To initialize the class helper at the beginning of each constructor of this class



/**
* The direction the spotlight is facing. The vector need not be normalized.

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

-> The direction vector of the spotlight. The vector need not be normalized.

/**
* The angle of the spotlight's inner cone. Surfaces whose angle to the light's origin is less than this angle
* receive the full light's intensity. At larger angles, the light intensity starts to drop. The valid range is
* {@code 0 <= innerAngle <= outerAngle}; values outside of this range can produce unexpected results.

This comment has been minimized.

@arapte

arapte Apr 16, 2021
Member

We should use Point or vertex instead of word surface. Using Point would be consistent with description of class.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 17, 2021

I'll fix the documentation once we establish how the light is rotated. I will need to add notes about the rotation transforms too.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 17, 2021

@kevinrushforth I have just tested on Ubuntu 20 and I don't see any issue (at least not something that's different than Windows). If it's an issue with the range property, does it happen with point lights too? Does it happen before this patch (but with the attenuation patch)? Also, does it happen when there is only 1 light turned on or also more (there are different files for the number of lights)?

If you can give the exact values for the properties that cause this I will be able to try a direct reproduction. I tested the animating sphere. I suggest to use the Boxes models since they are not animated and the range is constant.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 20, 2021

I'll try it on macOS with a non-retina display and see if that makes a difference (if so, it could point to a transform issue). I'll also try with the Boxes as you suggest.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented Apr 23, 2021

Yep, there's definitely a scale problem going on. It looks much different if I run it on a non-Retina display. And knowing that, I was able to reproduce it on Windows as well by comparing what it looks like with -Dglass.win.uiScale=1 and -Dglass.win.uiScale=2.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented Apr 23, 2021

I asked someone with a Mac Book Pro with a retina screen to run JavaFX 16, and I saw an issue with lighting positioning: the source of the light was not the position of the PointLight node. I couldn't debug it, but it seems that this issue is not related to this patch, and possibly not even to the range property specifically.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented May 13, 2021

Can we continue with the review? We need to decide about the rotation behavior and the performance options need to be tested.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented May 14, 2021

I started to look at this today, and will continue next week. There is also the outstanding issue with a retina display (scale=2) on macOS.

@nlisker
Copy link
Collaborator Author

@nlisker nlisker commented May 14, 2021

There is also the outstanding issue with a retina display (scale=2) on macOS.

Yes, it has to be looked at, but I think it is unrelated to this particular patch. If this is the case, we can file a different issue and continue with this one.

@kevinrushforth
Copy link
Member

@kevinrushforth kevinrushforth commented May 14, 2021

You may be right about the scale problem being preexisting. I'll double check when I test on Mac next week (both Retina and external screen).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants