[FEATURE] Add merge track option fast first pass

[mattdawkins]

[subdavis]

Questions for @mattdawkins

• Under what conditions should 2 tracks be considered merge-able?
• What if they have conflicting detections on the same frame? Should the order of selection for merge matter? Merge A with B means overwrite B with A, but Merge B with A means overwrite A with B?
• I assume this will mostly be used to join tracks after auto-tracking pipeline is run, so most of the time there shouldn't be the possibility of conflicts. Should we simply only allow merge for totally disjoint tracks?

[subdavis]

Here's how I think the UI could work:

• You enter merge mode by selecting a track like normal and then hitting some kind of merge button.
• All non-candidate merge tracks disappear (see above), and the track you selected turns a different color
• User has to use time nagivation controls to move forward and backward to select additional track(s) to add to the merge. Select by left click, and when selected, they turn a special merge color. Un-selected tracks remain their regular type-based color.
• In merge mode, there's a "finish merge" button when all tracks have been joined. This button brings up a summary of the merge, with details like
  • How many tracks are being merged
  • The frame span of the new track being created
  • The type of the new track (and maybe even an option to specify the type before you finalize)
  • The number of keyframes, interpolated frames, and empty frames that will be in the new track
  • A "CANCEL" button and a "MERGE" button.
• After a merge is completed, the merge mode turns off, the new merged track is selected regularly, and edit mode is always off.
• At this point, the user must press the save button in order to persist the merged track to the server (as with any other edit)

[mattdawkins]

This will mostly used to merge fragmented tracks (tracker lost a fish, and didn't pick it up when it re-emerged elsewhere) though I could forsee cases where there could be a couple overlapping states due to algorithm error. Don't know the best option for those cases, maybe just choose boxes from track 1 or average the boxes, wouldn't worry about it too much and just chose something. Another option would be to add a warning when this happens but that could also be annoying requiring multiple extra button presses.

Whatever the functionality is, we'd want to minimize button presses, e.g. select track 1, ctrl + select track 2 to highlight them both, right click + merge

[subdavis]

I expect this is one of the next highest priority tasks.

So far, I've done a few sample corrections, and most of the time the bad tracks are beyond repair. Someone who is familiar with the interpolation controls, most of the time, could build a new track from scratch faster than it would take to dissect a track that jumped across 3 targets (and also dissect several other tracks that jumped across overlapping targets) and rejoin the tracks, then go through and fill in the parts in the middle [citation-needed]

It's also pretty hard to keep track of what you have and have not corrected right now, so delete-and-recreate saves time on having to figure out which tracks you've already fixed.

Basically, I'm not sure yet that a simple multi-select-and-merge will be easy to use or even result in faster correction speed. I'm not aware of a any other annotation tool that has this capability, so doing something like this correctly will require extra care and research.

In order to do this properly, let's start by identifying some example datasets with merge-able tracks common enough that having this feature could hypothetically speed up annotation.

I'd actually like to try and come up with some way to quantify our results, such as identifying some small tasks where annotations take X minutes to correct with the current UI and Y minutes after the addition of some new merge-type tooling.

[BryonLewis]

I think one of the barriers is the temporal context. We don't have a good way to see into the past/future so you would have to do constant switching of frames and selecting between tracks for it to work. I think a concept of track tails or a visual indicator of boxes in the future/past would be needed before being able to make a track merge tool. Then you need to filter down that noise of all this temporal information to see only the tracks that you want to merge. The issue now becomes how complicated these tools are to use to filter in conjunction with merging the tracks.

I have some ideas but they still seem too complicated and they only fit the narrow scope of 'take beginning/end of one track and link it to the end/beginning of another track'. Nothing about overlapping frames requiring merging/splicing/cutting or other stuff like that.

[subdavis]

• How do you pick which parts of overlapping tracks to keep?
• What if one has a frame and another does not?
• We need track tails
• Can even show the mergeable tracks in the event viewer.