OpenGL mouse picking with ray casting

[kayomarz]

Branch opengl-mouse-picking-with-ray-casting has some code to experiement with with OpenGL mouse picking using ray casting.

How to test it out

1. Left click to select a single shape. In the case when multiple shapes lie in the path of the ray, then subsequent clicks select each shape in turn. This allows cycling amongst many closely overlapping objects. @kaveh this behaviour was arbitrarily done using blender as a reference - we can discuss and change it to suit to whatever works best for kons-9.

2. Left click while pressing the shift key to add an item to the selection.

3. To write custom selection behaviour, see test/demo-object-picking.lisp

TODOs:

1. triangles-list in file polyhedron.lisp will only work for convex polyhedrons but we need to be able to intersect for all cases.
2. Currently, we triangulate each shape on each pick. Try to re-use triangulation for subsequent picks if a shape has not changed.
3. Only tried with polyhedrons which have a triangulation routine. There might be other shapes which need to be dealt with?

How it tries to pick shapes

1. The first step in picking is to use glu:un-project to determine the far point under the mouse
2. Next, we extract the current camera position from the current modelview matrix using (gl:get-double :modelview-matrix)
3. The above two steps constitutes our ray from the camera to the far plane under the mouse
4. At this point, we create an instance of the origin library's origin.geometry.ray which gives us a ray object
5. Next we cycle through each objecet in the scene and do the following:
6. Get the axis aligned bounding box (aabb) of the shape in world coordinates
7. For the above aabb, we create an instance of the origin library's origin.geometry.aabb
8. Next we use the origin library function origin.geometry:raycast-aabb to intesect the above ray and aabb objects
9. If a shape's aabb did not intersect, we add it to a list called xs-miss and move on to the next shape.
10. If a shape's aabb does intersect, we further refine the intersection test by triangulating the shape and intersect the ray with each triangle for better certainty if the shape does intersect.
11. After intersecting each shape in the scene, we create a list called xs-hit containing a list of shapes sorted by distance from the camera. Anoter list xs-miss contains all shapes which did not intersect.
12. Finally we use xs-hit, xs-hit along with another list xs-current to decide which shapes should be selected in the scene. See test/demo-object-picking.lisp for more

Will look forward to comments and suggestions.

(Below is older stuff, not relevant anymore)

This demo picks objects whenever the mouse hovers over them which is done as follows:

1. Use a flag (*do-mouse-pick*) to detect mouse move
2. Immediately after each draw scene the object picker described above kicks in if the flag *do-mouse-pick* was set

### Picking resolution/accuracy

Since the ray is intersected with the aabb (axis aligned bounding box) an objects gets picked whenever it intersects the bounding box of a shape. As you will observe, this is not very accurate.

### Next steps

This was more like an fun experiment. If this type of picking might be of use to kons-9 (at-least until vulkan comes in), we can:

1. Improve the picking accuracy: If an object's aabb does get picked, attempt further intersections of the ray with the actual shape and not just the aabb
2. Design an api to enable/disable picking with useful options such as on mouse clicks / double clicks / hover, etc...

### Notes

1. For this demo, all objects that are in the path of the ray get picked, not only the closest one. This behaviour is easy to change and pick only the object closest to the camera. This can be done using the number returned by intersect funtion (origin.geometry:raycast-aabb) which indicates the distance from the camera.

Since this was an experiment, this is just a draft pull request to view code diffs and get feedback (rather than a normal pull request)

[kaveh808]

Nicely done. Looking forward to the suggested enhancements.

[aykaramba]

Just chiming in to say thank you for write-ups such as this. They serve as great learning material and high level insight into how these tests / features work.

Much appreciate the effort, love reading these.

[kayomarz]

Please review if these added functions are reasonable, the triangulation is based on existing code in the same file. Regarding naming of the added functions: since there was a face-points-array function which gets the face points, a similar nomenclature was used to name this function triangles-array which gets an array of triangles representing the polyhedron. Please suggest better names if any.

[kaveh808]

Looks reasonable. Just FYI the current triangulation expects convex polygons only. I don't currently handle concave polygons.

[kayomarz]

Thanks for taking a look. Have added a todo in the PR description above regarding the convex polygons precondition. Will address this, maybe in a subsequent PR once the basic object picking is done.

[kayomarz]

Just FYI the current triangulation expects convex polygons only. I don't currently handle concave polygons.

@kaveh808 I don't have much of a math background but after searching about triangulating concave polyhedrons, it seems non trivial, at least for me:) However if this could be useful to kons-9 roadmap, I can dig deeper into it. btw I came across CGAL which might do the triangulation we need. Could CGAL be useful? As you rightly mentioned elsewhere its also important to keep very few dependencies and in which case it might be better to postpone the use of such a library until we really need it at a later stage?

[kayomarz]

Updates have been commited to branch opengl-mouse-picking-with-ray-casting:

Try the updates by getting the latest of the branch and as before, after (run), evaluate (add-demo-shapes-to-scene) in the REPL to add a few octahedrons to the scene.

What to look out for in this update:

Hover the mouse over shapes to begin picking them - You will observe that the shape now gets selected only when the mouse hovers its geometry and not just its aabb

Update summary for commits made until now:

1. Since the origin library does not have a function to compute ray - triangle intersection and to avoid adding external library dependencies, we have ported a simple but reliable C function (MIT Licensed) which seems to work. Links to more information as well as the license are in the lisp file here.
2. For better picking accuracy, once a ray intersects a shape's aabb we further intersect the ray with triangles from triangulating a polyhedron to test if the ray intersects with a shapes geometry.

Questions:

1. Where should src/graphics/object-picking.lisp actually reside? (It is independent of glfw and opengl)
2. Please can someone review functions added to polyhedron.lisp which help to get hold of triangles representing the polyhedron after triangulation.

Observations:

Near the center of the viewport, picking seems reasonably precise. As we moving towards the periphery of the viewport a small but noticeable offset begins to grow between the mouse position and the shape which gets selected. I wonder if this unwanted behaviour could be related to perspective of the perpective camera 🤔

update: this unwanted behaviour is now fixed, which was due to a wrong function being used to instantiate a ray. see #234

[kayomarz]

Update: You can now select a shape using left-click or multiple shapes using the shift key modifier with left-click to select multiple shapes. To write a custom selection function, see test/demo-object-picking.lisp

[kayomarz]

@kaveh808 would be great to get your feedback on the object picking - It is ready for review. Thanks!

[kaveh808]

Approving. Thank you for such a useful enhancement.

[kayomarz]

I enjoyed working on it :) I plan to do more random stuff with kons-9. If you have any specific tasks in mind, let me know.

[kaveh808]

Great, @kayomarz :)

It would be helpful to implement OpenGL texture mapping.

Currently we do basic OpenGL rendering and viewing, without explicit camera objects or materials. Implementing simple texture mapping will allow us to enhance both our 3D display and support 2D GUI icons.

I have been holding off implementing materials, lights, and cameras until we figure out our rendering backend.