Prefer user instances when "Copy Prior Frame"

[Deruijter]

Hi,

I'm fairly new to SLEAP so I'm not sure I'm doing something incorrectly or that this is unintended behavior.

I've manually labeled about 240 frames (~480 animals) in my videos. I'm now testing to run inference on unlabeled video frames. I'm starting with predicting an individual frame that takes place after a frame that was: first predicted, and then manually corrected.

I was not satisfied with the inferred frame as the positioning was more off than I expected (particularly because the animal hardly moved). I then proceeded using the "copy prior frame" option to label the frame myself. To my surprise the new instance was different than that of the prior frame. It actually has the same shape as the original prediction of the prior frame, but not the corrected instance. I'm guessing the same is happening while trying to infer this frame.

Is it correct that "post prediction -corrected- instances" are somehow ignored in this regard? I was hoping to kick start the inference in the correct direction by starting with a few manually corrected frames.

Note that predictions + corrections of these frames were done in version 1.2.5. I now updated to 1.2.9 but the "copy prior frame" problem seems to persist.

Some images to show what is going on:
Predicted instance:
https://img.onl/AEFAqj

After prediction, I corrected one of the nodes (posterior right leg):
https://img.onl/8ls4Px

I then moved to the next frame, and created a new instance by right clicking and selecting "Copy Prior Frame"
https://img.onl/R7rsE
Here, the posterior right leg is in the uncorrected position.
My expectation was that the instances on the last two figures would be identical.

[roomrys]

Thanks for reporting @Deruijter,

A bit of strange logic here (I'll need to double check why this is the case)... It seems SLEAP checks if the number of instances in the previous frame is greater than the number of instances in the current frame and uses the previous frame instance at index equal to the number of instances in the current frame.

sleap/sleap/gui/commands.py

Lines 2576 to 2603 in 19b9503

	if (
	init_method == "best" and copy_instance is None
	) or init_method == "prior_frame":
	# Otherwise, if there are instances in previous frames,
	# copy the points from one of those instances.
	prev_idx = cls.get_previous_frame_index(context)
	if prev_idx is not None:
	prev_instances = context.labels.find(
	context.state["video"], prev_idx, return_new=True
	)[0].instances
	if len(prev_instances) > len(context.state["labeled_frame"].instances):
	# If more instances in previous frame than current, then use the
	# first unmatched instance.
	copy_instance = prev_instances[
	len(context.state["labeled_frame"].instances)
	]
	from_prev_frame = True
	elif init_method == "best" and (
	context.state["labeled_frame"].instances
	):
	# Otherwise, if there are already instances in current frame,
	# copy the points from the last instance added to frame.
	copy_instance = context.state["labeled_frame"].instances[-1]
	elif len(prev_instances):
	# Otherwise use the last instance added to previous frame.
	copy_instance = prev_instances[-1]
	from_prev_frame = True

The last conditional would give us the result we want in this case (using the last instance added in the previous frame), but we never reach that bit of code.

Regardless, we should be preferring user instances over predicted instances or at least check whether the grabbed instance has an associated user instance.

sleap/sleap/instance.py

Lines 1542 to 1563 in 19b9503

	@property
	def instances_to_show(self) -> List[Instance]:
	"""Return a list of instances to show in GUI for this frame.
	This list will not include any predicted instances for which
	there's a corresponding regular instance.
	Returns:
	List of instances to show in GUI.
	"""
	unused_predictions = self.unused_predictions
	inst_to_show = [
	inst
	for inst in self._instances
	if type(inst) == Instance or inst in unused_predictions
	]
	inst_to_show.sort(
	key=lambda inst: inst.track.spawned_on
	if inst.track is not None
	else math.inf
	)
	return inst_to_show

[Deruijter]

Hi @roomrys, thanks for your fast response! If I understand correctly, the predicted instances are still stored somewhere, so the "corrected predicted" instances is actually a new instance? (just asking because it's good to understand how things work under the hood) :)

With respect to my other comment, i.e. trying to "kickstart" the inference using user predicted frames. Is something like this possible? I feel like if the initial frames are not entirely correct, the following inferred frames will likely have the same issues. (or should I ask this question somewhere else?)

[roomrys]

Hi @Deruijter,

Always good to know how things work! Yes, when a user creates a user instance from a predicted instance, the predicted instance is still stored, but not displayed. The user instance is a new instance.

I'd like to clarify that training only uses user-labeled frames as training frames. The predicted frames/instances will not be used as training examples (until the user converts them to user instances - or more accurately creates user instances from these predictions). If the user-labeled instances are not correct, then you will get incorrect prediction. However, the incorrect predictions will not affect new predictions as they will not be used to train the model. The predictions serve to show how the current model is performing and which poses might need more user-labels/training examples.

Let me know if you have any more questions. If you feel we are steering off topic, definitely feel free to create a new discussion.

Thanks,
Liezl