Add flythrough animation feature

[andinet]

We need a flythrough animation feature in VPAW. This is to aid in communicating aspects of a patient’s airway/anatomy to other clinicians. A flythrough path is defined by creating a set of keyframes that store camera poses. The camera poses are interpolated between keyframes to produce smooth camera movement along the camera path.

Task:

A standalone KeyFrameEditor module (or feature added to an existing module) for generating and exporting volume rendering animations. The module will provide a timeline and tracks allowing to save, edit and delete key frames associated with transfer function, ROI clipping box and camera position and orientation. Available keyframe interpolation functions will include ramp, exponential and bezier. Camera path will be represented by a curve annotation.

The plan is to distribute this feature as a standalone module as a Slicer extension or added to an existing module in Slicer core.

PS

Related modules to explore

1. Sequence module ( https://slicer.readthedocs.io/en/latest/user_guide/modules/sequences.html )

2. ScreenCapture module ( https://slicer.readthedocs.io/en/latest/user_guide/modules/screencapture.html )

3. Endoscope module ( https://slicer.readthedocs.io/en/latest/user_guide/modules/endoscopy.html)

4. SlicerMorph Animator module (Animator:

[jadh4v]

Initial survey of modules / extensions that can support our Flythrough requirements.
WIP - will be updating this post more with additional info / discussions (fyi: @jcfr @andinet ):

• Sequences module
  Documentation: https://slicer.readthedocs.io/en/latest/user_guide/modules/sequences.html
  Source code: https://github.com/Slicer/Slicer/tree/master/Modules/Loadable/Sequences

  • Too general? Hard to use.
  • Screen Capture is based on Sequences, so perhaps we can extend on it instead or look at how it uses Sequences for lessons / good approach

• ScreenCapture module
  Documentation: https://slicer.readthedocs.io/en/latest/user_guide/modules/screencapture.html
  Source code: https://github.com/Slicer/Slicer/tree/master/Modules/Scripted/ScreenCapture

  • Currently works in tandem with Sequence module (complexity) or 3D rotations
  • No live preview or visualization (directly generates video / images)
  • We can add FiducialMarkups as additional animation mode (will require camera pos info on point lists)

• Endoscopy
  Documentation: https://slicer.readthedocs.io/en/latest/user_guide/modules/endoscopy.html
  Source code: https://github.com/Slicer/Slicer/tree/master/Modules/Scripted/Endoscopy

  • Very easy to use
  • Provides live preview / visualization
  • Can create a path using points and flythrough animation
  • Cannot attach/modify camera poses (camera always looking along the path).
  • We can add camera poses as markups to interpolate.
  • Will need to add keyframe interpolation algorithm(s).

• SlicerMorph Animator
  Documentation: https://github.com/SlicerMorph/SlicerMorph/tree/master/Docs/Animator
  Source Code: https://github.com/SlicerMorph/SlicerMorph/tree/master/Animator

  • A basic keyframe-based animation of 3D volumes.
  • Supports interpolation of regions of interests (ROI), rotations, and transfer functions (volume property) for volume rendering.
  • Output can be either as an image sequence of frames or can be compiled into mp4 format.
  • Currently doesn't support following paths / curves but appears to already have keyframe interpolation. It is based on ScreenCapture.
  • Should investigate how easy it would be to add camera path and interpolation using camera-pose keyframes.

• Volume reslicing
  https://discourse.slicer.org/t/can-i-make-igt-volume-reslice-driver-run-automatically-after-an-event/16215

• CameraPath
  Source code: https://github.com/KitwareMedical/Slicer-CameraPath
  Documentation: https://www.slicer.org/wiki/Documentation/Labs/FlyThroughNavigation

  • Based on code review, it might have most of the desired functions already implemented. Existing components:
    • vtkMRMLCameraPathStorageNode: MRML node for camera path storage
    • vtkMRMLCameraPathNode: MRML node to hold the information about a camera path
    • vtkMRMLPointSplineNode: MRML node to hold the information about a 3D spline
  • Faced some build errors due to old Slicer template. Might need an update of latest Slicer App template.

[jadh4v]

See last paragraph in comment here: https://discourse.slicer.org/t/support-for-keyframe-based-animation/6885/6

How should we implement vpaw flythrough? Should it be keyframe interpolation of camera poses along a curve, or features/structures to look-at as you flythrough.

[jadh4v]

Based on our internal discussions and from the discourse discussion thread, we have decided to modify the Endoscopy module to provide full support for a camera flythrough and keyframe interpolation. We can borrow specific implementations from CameraPath as/if needed.

Keyframe Interpolation

(From Discourse)

• Add some pure-VTK camera interpolation functionality, perhaps just one class based on vtkQuaternionInterpolator with some extra code to handle changes in field of view and maybe other parameters. This could go in vtkAddon if it’s C++ or in Endoscopy logic if it could be done in python (I’d think it could).

User Interface

(From Discourse)

• We could use the 3D view camera state itself as a way to insert keyframes. Initially we were considering having a flight mode for the 3D view to assist in flying around and placing keyframes, but since we can define the path curve first, it is probably not needed. The user can just fly along the path and stop and insert keyframes wherever necessary.

• Regarding display of keyframes we can simply use a camera mesh model to indicate the position or use Plane markups

• Along with camera pos and fov, would it make sense to also interpolate volume property, etc for better local control of visualization?
  The Animator module has some code for this and it would be nice to either reuse or generalize this feature. Currently it requires that the transfer functions have the same number of control points at the keyframes.

• We should provide headlights that move with the camera (endoscopy simulation).

Use of Markups

(Discussions with @jcfr)

• Plane markup has controls to work as a camera pose. Can be used as an interaction widget for placing and modifying camera poses along the flythrough curve.
• Explore use of Dependant markups. It may also be worth looking into Dependant Markups for automatic updating of related markups.
• Markup Constraints by @allemangD [David Allemang] will be useful to sync camera position/orientation markups with the path curve and vice-versa:
  https://projectweek.na-mic.org/PW37_2022_Virtual/Projects/MarkupConstraints/

Nice To Have

(From Discourse)

• It would also make sense for the Endoscopy module to export to a Sequence, like the Animator does, so that it could be recorded by the ScreenCapture logic. -- Steve Pieper

[allemangD]

It may also be worth looking into Dependant Markups for automatic updating of related markups.

MarkupConstraints now has its own repository; refer to that instead of the old DependantMarkup branch which will no longer be updated. https://github.com/KitwareMedical/SlicerMarkupConstraints

I plan to implement more adaptors (and create a system to register custom adaptors) to allow the constraints system to observe/constrain other nodes than markup control points; for example to observe a spline in its entirety rather than a single control point, or to constrain the position of a model or volume. From what I gather here, that feature would be important for synchronizing camera orientation with other control points so I will create a relevant issue and prioritize that work.

Plane markup has controls to work as a camera pose. Can be used as an interaction widget for placing and modifying camera poses along the flythrough curve.

I believe all markup nodes can enable the interaction widget; for camera position it may be better to use a fiducial markup with a single oriented point. I can confirm the interaction widget functions well with MarkupConstraints.

[jadh4v]

Since we want to upgrade endoscopy and contribute it upstream into core Slicer, I have created a parallel issue / enhancement request in Slicer core here:
Slicer/Slicer#6502
cc: @HarryDC @andinet

[Leengit]

I am closing this issue in favor of Issue Slicer/Slicer#6502 under the thinking that the implementation will be within Slicer's Endoscopy module rather than within VPAW.