Empty packet sent before correct packet on output streams for some frames, causing 1-frame offset.

[megawattfs]

Plugin Version or Commit ID

v0.5.0

Unity Version

2020.3.23f1

Your Host OS

Windows 10

Target Platform

Windows Standalone

Description

I am using MediaPipe Pose on a series of images. I pass input into the "input_video" stream, then wait for an output packet in the "pose_landmarks" stream before providing the next input packet. Therefore, there should be no race conditions between input packets. I have also verified that no input packets are being assigned identical timestamps.

I had the observeTimestampBounds option set to "true" on all four output streams (pose_detection, pose_landmarks, pose_world_landmarks, and roi_from_landmarks) so that an empty packet would be sent in case detection failed. In our data, there are no input images where pose detection fails completely. However, occasionally an empty packet appears in the "pose_landmarks" output stream after an image is input. Whenever this happens, the correct packet for that image does appear in the "pose_landmarks" output stream, but not until an additional input (i.e. the next image frame) has been input. Consequently, every frame input thereafter will be incorrectly associated with the landmarks from the previous frame: There is an offset of 1 frame for each time an empty packet appears in "pose_landmarks."

Here is a simplified example of what happens when this situation occurs:

1. Frame X sent to "input_video"
2. Retrieve empty packet from "pose_landmarks"
3. Frame X+1 sent to "input_video"
4. Retrieve packet containing Frame X's landmarks from "pose_landmarks"
5. Frame X+2 sent to "input_video"
6. Retrieve packet containing Frame X+1's landmarks from "pose_landmarks"
   ...etc. each frame will be offset by one, and there is no way to distinguish an empty packet sent due to failed pose detection from an empty packet sent between two consecutive valid frame landmark packets as shown above.

If I set the observeTimestampBounds option to "false," the problem is eliminated; however, this is problematic because an empty packet is never sent to indicate that pose detection has actually failed on a frame, which may be required for other datasets. Changing the timeout value for the GraphRunner (PoseTrackingGraph) up to 5 seconds did not resolve the issue.

Additional Context

I already verified that no two input packets have the same timestamp. I also tried increasing the timeout time on the GraphRunner. As described above, setting observeTimestampBounds to false for the output stream fixes the problem but will cause the program to hang if I try to pass in data for which pose detection fails. The issue occurs regardless of whether landmark smoothing is enabled.

[homuler]

v0.5.0

Is this correct? If so, please upgrade to v0.9.1 first.

Changing the timeout value for the GraphRunner (PoseTrackingGraph) up to 5 seconds did not resolve the issue.

That's because GraphRunner in the version you're using did not respect the timeout settings in synchronous mode.
Please check if it's resolved at v0.9.1.

Whenever this happens, the correct packet for that image does appear in the "pose_landmarks" output stream, but not until an additional input (i.e. the next image frame) has been input.

I think this is not necessarily true.

For example, the following code will retrieve the next value from the pose_landmarks stream, but it does not mean that there's only one value in the pose_landmarks stream.

_poseLandmarksStream.TryGetNext(out var poseLandmarks);;

In past versions (maybe including the version you're using), there was a method (TryGetLatest) to retrieve the latest value.

MediaPipeUnityPlugin/Assets/Mediapipe/Samples/Common/Scripts/GraphRunner.cs

Lines 427 to 447 in de44dac

	public bool TryGetLatest(out TValue value)
	{
	if (HasNextValue())
	{
	var queueSize = _poller.QueueSize();
	// Assume that queue size will not be reduced from another thread.
	while (queueSize-- > 0)
	{
	if (!Next())
	{
	value = default;
	return false;
	}
	}
	value = _outputPacket.Get();
	return true;
	}
	value = default;
	return false;
	}

I thought the problem this solves could also be solved by setting a timeout, so I removed it.
If this assumption is incorrect and there's a situation where it's necessary, please let me know.

setting observeTimestampBounds to false for the output stream fixes the problem but will cause the program to hang if I try to pass in data for which pose detection fails.

There's a way to set observeTimestampBounds to false and still prevent the main thread from hanging.
If interested, check out the Non-Blocking Sync mode in the latest version.

[megawattfs]

Whenever this happens, the correct packet for that image does appear in the "pose_landmarks" output stream, but not until an additional input (i.e. the next image frame) has been input.

I think this is not necessarily true.

The reason I believe there is only one value in the stream is because if I call TryGetNext() a second time after retrieving the empty packet but before providing another input, it hangs indefinitely. However, if I call TryGetNext() after providing another input, it retrieves the previous input's landmarks.

I will try the newer version and see if it resolves the problem.

[homuler]

Will you share the input video to reproduce this issue?

[megawattfs]

No, sorry, the data is not public. Detection does not fail on any frames in the dataset individually, though.

[homuler]

OK, then will you tell me why you think there is an offset of 1 frame?
Since MediaPipe uses the previous result to localize the landmarks on the current image, I think you can't assert that just because the results are not accurate.

[megawattfs]

Because the input is not a high frame rate video (I have landmark smoothing turned off for this reason). Each frame is significantly different from the last, and each set of landmarks very closely matches the pose for the previous frame (the frame that first sent an empty packet) every time this happens. The poses are very easily distinguishable and recognizable (not generic) and the output for all subsequent frames matches the previous frame every single time an empty packet is sent.

[homuler]

Each frame is significantly different from the last

Does this mean that you are using a video that is a series of still images joined at intervals of a few frames?
I think this can be problematic as mentioned in #552 (comment).

cf. https://google.github.io/mediapipe/solutions/pose.html#ml-pipeline

Note that for video use cases the detector is invoked only as needed, i.e., for the very first frame and when the tracker could no longer identify body pose presence in the previous frame. For other frames the pipeline simply derives the ROI from the previous frame’s pose landmarks.

By the way, I don't have any data where the detection succeeds for all frames but returns empty packets when observeTimestampBounds is set to true (my data won't return empty packets).

Not sure if my understanding is correct, but I thought MediaPipe would not return an empty packet in synchronous mode if observeTimestampBounds was set to true and the detection should succeed for that input image (I thought it would block the thread until it was clear whether the detection would be successful or not).
Thus, it is not clear to me what is happening without checking such data.

[megawattfs]

Does this mean that you are using a video that is a series of still images joined at intervals of a few frames?
I think this can be problematic as mentioned in #552 (comment).

We are in image mode, not video, so I don't think this should be a problem. It works on still images and series of still images without issue most of the time.

Not sure if my understanding is correct, but I thought MediaPipe would not return an empty packet in synchronous mode if observeTimestampBounds was set to true and the detection should succeed for that input image (I thought it would block the thread until it was clear whether the detection would be successful or not).

I think it actually does the opposite: It allows the calculator to time out so that it stops waiting for a packet that will never arrive.

I discovered that we are on version 0.8.1, not 0.5.0. Was the timeout issue fixed in 0.8.1? I cannot find any references to it in the changelog. I will update to 0.9.3 and check if that fixes the problem as soon as I can.

[homuler]

We are in image mode, not video

What does "image mode" mean? Is it that you set use_prev_landmarks to false manually?

I think it actually does the opposite: It allows the calculator to time out so that it stops waiting for a packet that will never arrive.

Have you read the source code?
https://github.com/google/mediapipe/blob/4a20e9909d55838d5630366ce719844cf06ae85c/mediapipe/framework/graph_output_stream.cc#L177-L222

If so, will you tell me why you think so?

Was the timeout issue fixed in 0.8.1?

No.
But first, please check if it occurs when using the official C++ code.
If it occurs, then this is not the plugin's problem, so please ask it in the official community.

[homuler]

Now I think that it occurs as described in #642 (comment).

[megawattfs]

Thanks for bringing this to my attention (the author of that issue was a summer intern at my company, so I am picking up where he left off on this issue.) I'll be sure to test this workaround on our data when time allows. Best, Margaret

…

On Sat, Aug 13, 2022 at 12:08 AM Junrou Nishida ***@***.***> wrote: Now I think that it occurs as described in #642 (comment) <#642 (comment)> . — Reply to this email directly, view it on GitHub <#552 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK5MFY74YUTS3DPPOHBEUN3VY4NU5ANCNFSM5VFDX3RQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Margaret Schrayer ***@***.*** 609-480-6272