Create a specification for video annotation

[mzur]

Examine existing tools for video annotation and compile a collection of their features. Then organize a workshop with MI2 participants to agree on a specification an feature requirement for a video annotation tool. Important questions are:

• Is an annotation only valid for a single video frame? Or is it valid for a single "object" across multiple frames (i.e. can it move)?
• ...

[mzur]

Example tools:

• Video Annotation and Reference System (VARS)
• Computer Vision Annotation Tool (CVAT)
  This tool supports automatic "interpolation" of annotation across several video frames (an annotation "track"). Their idea of several manual "key frames" for the annotation might be interesting. You can also combine "tracks" which are multiple annotations (where an annotation spans several frames) of an object, that disappears and appears again in the video.
• Microsoft VoTT uses the Camshift tracking algorithm.
• The Annostation Labeling System uses an interface for the annotation overview similar to what I had in mind:
  
  At the bottom you can see the "tracks" for each annotation that span multiple image frames. The frame preview thumbnails may be useful as well (similar to video editing software).
• VIDLIB Deep-Sea Video Platform
• Adelie fir ArcGIS
• Anvil Video Annotation Research Tool
• VideoAnt
• Ocean Floor Observation System (OFOP).
  Abstract, poster. We can also ask Timm for a demo version. This tool supports live streaming of video data from a deployed vehicle. IIRC it also allows "snapshotting" of time points of the live video for later, more detailed annotation.

[mzur]

Tim had a comment about the timeline containing the "tracks" of the annotations (see above). Video editing software usually allows zooming in on the timeline to focus on a specific timespan in the video. This is realized by stretching the timeline and pushing the ends left and right out of the viewport. Tim said that the global context is too important in our scenario so we should not implement zooming like that. Instead, we should implement it like a "fish eye" view where the zoomed-in part is magnified and everything else is squashed but still visible in the periphery. Maybe we can offer both variants later but for our prototype we should focus on the more important one.

To fully make use of the global context, tracks of annotations with the same class should be highlighted when one is hovered in the video or in the timeline.

[mzur]

Timm said a nice feature would be live stream support. Users can watch the video, hit pause, rewind etc. like a TV/YouTube live stream. While the stream is running, users should be able to make "snapshots" or "bookmarks" for specific time points in real time so they can revise their bookmarks for later more detailed annotation.

[mzur]

Tim said if would be OK if I implemented an early propotype of the video annotation tool before we have a meeting/workshop to agree on a specification. The propotype is intended for use during the SO268 cruise and arranging a meeting before the cruise is probably not possible. I'll concentrate on this feature set:

• Web based application
• Video playback
• "Snapshot" annotations which mark a single frame of the video
• "Track" annotations which mark (moving) locations over several video frames
• Utilitarian UI
• (nice to have) Video stream support
• (nice to have) Automatic object tracking for new "track" annotations

[mzur]

A first prototype is implemented as described here. It has all features except the (nice to have) from the comment above. These are continued in their own issues (#6, #7).

[mzur]

The first version of the prototype is ready and will be deployed during SO268. We can compile a best practice specification from the lessions learned there and from feedback of users of our BIIGLE instance. Continued in #3.