Upgrade endoscopy to support arbitrary camera poses and key frame interpolation

[jadh4v]

Is your feature request related to a problem? Please describe.

We need a flythrough animation feature in VPAW. This is to aid in communicating aspects of a patient’s airway/anatomy to other clinicians. Current Endoscopy module comes very close to satisfying our requirements. However, it has some missing feature such as:

• Cannot attach/modify camera poses (camera always looking along the path).
• Needs keyframe interpolation algorithm(s).

Describe the solution you'd like

Description of suggested solution copied / referenced from the posted comment here.

A flythrough path is defined by creating a set of keyframes that store camera poses. The camera poses are interpolated between keyframes to produce smooth camera movement along the camera path.

Based on our internal discussions and from the discourse discussion thread, we have decided to modify the Endoscopy module to provide full support for a camera flythrough and keyframe interpolation. We can borrow specific implementations from CameraPath¹ as/if needed.

Keyframe Interpolation

(From Discourse)

• Add some pure-VTK camera interpolation functionality, perhaps just one class based on vtkQuaternionInterpolator with some extra code to handle changes in field of view and maybe other parameters. This could go in vtkAddon if it’s C++ or in Endoscopy logic if it could be done in python (I’d think it could).

User Interface

(From Discourse)

• We could use the 3D view camera state itself as a way to insert keyframes. Initially we were considering having a flight mode for the 3D view to assist in flying around and placing keyframes, but since we can define the path curve first, it is probably not needed. The user can just fly along the path and stop and insert keyframes wherever necessary.

• Regarding display of keyframes we can simply use a camera mesh model to indicate the position or use Plane markups

• Along with camera pos and fov, would it make sense to also interpolate volume property, etc for better local control of visualization?
  The Animator² module has some code for this and it would be nice to either reuse or generalize this feature. Currently it requires that the transfer functions have the same number of control points at the keyframes.

• We should provide headlights that move with the camera (endoscopy simulation).

Use of Markups

(Discussions with @jcfr)

• Plane markup has controls to work as a camera pose. Can be used as an interaction widget for placing and modifying camera poses along the flythrough curve.
• Explore use of Dependant markups. It may also be worth looking into Dependant Markups for automatic updating of related markups.
• Markup Constraints by @allemangD [David Allemang] will be useful to sync camera position/orientation markups with the path curve and vice-versa:
  https://projectweek.na-mic.org/PW37_2022_Virtual/Projects/MarkupConstraints/

Nice To Have

(From Discourse)

• It would also make sense for the Endoscopy module to export to a Sequence, like the Animator does, so that it could be recorded by the ScreenCapture logic. -- Steve Pieper

Describe alternatives you've considered

Considered alternatives and some pros / cons identified in each are documented here: KitwareMedical/vpaw#9 (comment)

Additional context

While this issue has already been described in detail on the VPAW project here, I am creating a Slicer issue since we hope to contribute upstream / upgrade the official Endoscopy module. We believe that the Endoscopy module, in its current form, is not as useful for virtual endoscopy animation due to the restricted camera movement.

Footnotes

1. https://github.com/KitwareMedical/Slicer-CameraPath ↩

2. https://github.com/SlicerMorph/SlicerMorph/blob/master/Animator/Animator.py#L277-L434 ↩

[Leengit]

I am looking to modify Slicer/Modules/Scripted/Endoscopy/Endoscopy.py to support this functionality. First question: We want keyframe camera orientation to be relative to the current Endoscopy module behavior, right? For example, consider a roller coaster. The Endoscopy module currently has the camera always looking forward, which means down the slope on downslopes and up the slope on upslopes. If someone gives us two key frames that are a forward-looking, horizontal camera orientation at two consecutive peaks of the roller coaster, we want to interpret them as forward-looking and give the current Endoscopy behavior, rather than interpreting them as horizontal-looking and having the module's new behavior be to always look horizontally when between those peaks, even on downslopes and upslopes.

That is, we want the new functionality to be a transformation that is applied after the current set of transformations, rather than instead of the current transformations, right? It would be only this relative-to-Endoscopy's-current-behavior transformation that we'd interpolate with quaternion slerp or similar. Likewise, if we support it, adjustments to focal length or zoom would be relative to Endoscopy's current behavior.

[jcfr]

We want keyframe camera orientation to be relative to the current Endoscopy module behavior, right?

Starting to update this module makes sense. As we move forward, we may have to improve/refactor some of the underlying libraries.

[pieper]

Glad you are working on this @Leengit.

This was the first Python ScriptedModule so it definitely is due for an update : D

But even before that there was an even more sophisticated endoscopy module in slicer2 (in Tcl of course) from Delphine Nain that is described in this paper and you can also find her thesis on the topic if you search around. She implemented a lot of good features for this task.

[lassoan]

Markups curve provide a robust and smooth curve coordinate system (tangent and normal direction for each curve point), so it makes sense to use this and apply any specify camera pose in this coordinate system. By default, camera can be forward-looking but the user could move it around (rotate and slightly move, maybe also adjust the field of view, speed of motion).

Keyframe data could be stored in the curve node itself: for each control point as static measurement. Or it could be stored in a sequence, using distance along the centerline as index. Linear interpolation probably works for position, zoom, speed; and slerp for orientation.

To adjust the pose of the camera for each key frame, we should re-map the camera manipulation keyboard and mouse gestures so that the when you move the camera forward/backward it moves the camera focal point forward/backward. There is an example in the script repository for this. We should also make the camera focal point setting more discoverable (it is a right-click menu action in recent Slicer versions), maybe with a movable markup point.

[HarryDC]

@jcfr I think we lost track of this in the handover between me and @jadh4v have a look, this was still work in progress when i stopped 3be9e40 i think it stopped being a priority

[HarryDC]

Iirc it does use markups to let the user orient the camera, but in working at it, i wasn't sure if a second mode that just describes the camera view point (being one, or a curve as well) wouldn't be easier to control for the user

[jadh4v]

Default behavior: I think it makes sense to have the forward-looking current approach be the default and key frames only intermittently influencing the camera pose. However, it would also be useful to provide a way to control how far the influence of a keyframe goes, perhaps using two weights (approach and departing).

key-frame editing: I am hoping that we could include both, third-person as well as first-person key frame manipulation as both approaches could be useful. Either way, we might have to show the result of third-person manipulation in a small window/frame, which could be made interactable as first-person manipulation???

[jcfr]

this was still work in progress

Thanks for pointing this out 🙏

For reference:

• ENH: Edit orientations in flythrough (DRAFT) #6541

third-person and first-person

We could leverage dual 3d view or having a dedicated layout

Keyframe data could be stored in the curve node itself: for each control point as static measurement.

Ditto. Working on hardening the infrastructure and how the information is stored/organized/serialized should be done first.

[jcfr]

we should re-map the camera manipulation keyboard and mouse gestures so that the when you move the camera forward/backward it moves the camera focal point forward/backward. There is an example in the script repository for this.

https://slicer.readthedocs.io/en/latest/developer_guide/script_repository.html#customize-keyboard-mouse-gestures-in-viewers

[Leengit]

However, it would also be useful to provide a way to control how far the influence of a keyframe goes, perhaps using two weights (approach and departing).

The implementation in #7165 uses quaternion slerp to interpolate between keyframes. If an effect is to start at some frame, reach a maximum value, and then end at some frame, that is entered by setting the first and third of these keyframes to have identity (unchanged) orientation, and setting the second keyframe to have the desired excursion. The remaining (non-key) frames from the first of these to the third of these will have their orientations computed via interpolation.

[Leengit]

I am hoping that we could include both, third-person as well as first-person key frame manipulation as both approaches could be useful.

The implementation in #7165 supports first-person keyframe manipulation.

[Leengit]

To adjust the pose of the camera for each key frame, we should re-map the camera manipulation keyboard and mouse gestures so that the when you move the camera forward/backward it moves the camera focal point forward/backward. There is an example in the script repository for this.

We are supporting new camera orientations (pitch, roll, yaw) in #7165, but not other degrees of freedom that a camera could have. The latter are deferred for now.

[Leengit]

Should we close this as completed by Pull Request #7165? If not, let's re-iterate, in fresh comments here, those aspects of this issue that are outstanding.