[QUESTION] kvssink keeps dropping frames from an rtsp source

[floatingpurr]

I'm trying to stream to KVS several rtsp flows (video only).

I built my environment exactly how it is described here: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp#build. However, kvssink keeps dropping frames with this message:

DEBUG - Dropping frame with flag: 1536

and the KVS stream remains empty.

Case 1: foo (not working)

Here is my command for the camera foo:

$ gst-launch-1.0 -v rtspsrc location="rtsp://user:pass@foo" short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name="YourStreamName" storage-size=128 access-key="aaa" secret-key="aaa" aws-region="eu-central-1"

and here is the output

Here is the fakesink:

$ gst-launch-1.0 rtspsrc location="rtsp://user:pass@foo" ! rtph264depay ! capsfilter caps="video/x-h264" ! fakesink

here is the output

Even in VLC, the video is properly streamed.

Case 2: bar (working!)

I tried the same command with another camera (bar) and in that case, all the things work:

$ gst-launch-1.0 -v rtspsrc location="rtsp://user:pass@bar" short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name="YourStreamName" storage-size=128 access-key="aaa" secret-key="aaa" aws-region="eu-central-1"

and here is the output

Case 3: baz (not working, again)

Strangely enough, if I try with the camera baz (same model of camera bar), things do not work again (dropped frames).

$ gst-launch-1.0 -v rtspsrc location="rtsp://user:pass@baz" short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name="YourStreamName" storage-size=128 access-key="aaa" secret-key="aaa" aws-region="eu-central-1"

here is the output

Here is the fakesink for this case

$ gst-launch-1.0 -v rtspsrc location="rtsp://user:pass@baz" short-header=TRUE ! rtph264depay ! h264parse ! fakesink

and here is the output

Again, baz streams properly to VLC.

Unfortunately, I can't access to any camera control panel. I can just consume the rtsp streams as they are.

I'd kindly ask you some help on how to troubleshoot this kind of problem.

[hassanctech]

The frame dropping message should be benign. Here are the frame flags:

https://gstreamer.freedesktop.org/documentation/gstreamer/gstbuffer.html?gi-language=c#GstBufferFlags

So 1536 = 1024 + 512 meaning both GST_BUFFER_FLAG_HEADER and GST_BUFFER_FLAG_MARKER are set, and 1600 = 1024 + 512 + 64 meaning GST_BUFFER_FLAG_HEADER, GST_BUFFER_FLAG_MARKER, GST_BUFFER_FLAG_DISCONT are set. These will be dropped but should not impact the stream, in fact I see this happens in the bar case which seems to be working fine.

I did notice this in your logs:

Could not receive any UDP packets for 5.0000 seconds, maybe your firewall is blocking it. Retrying using a tcp connection.

If you know you will not support UDP in order to avoid this latency you can set this property: https://gstreamer.freedesktop.org/documentation/rtsp/rtspsrc.html?gi-language=c#rtspsrc:protocols to only tcp because if you leave it to the default it will always try UDP first which will incur latency for you.

In theory fixing the above to TCP only with your network configuration shouldn't fix the issue but I wouldn't be surprised if there were some extra issues that show up when you're renegotiating the rtsp stream the camera so I would definitely suggest to make this change if you know you will not support UDP.

This likely will not fix your issue but I see your terminate the pipeline in the foo case after 13s. Can you please try to run it for 1-2 minutes and see if we can get some error codes returned from the SDK to be printed in the logs?

[floatingpurr]

Thank you @hassanctech for your suggestions.

Here is the longer log for the camera foo:

foo.log

And here is the same log with the property protocols=tcp

foo_tcp.log

Plese note that in both cases I get in Kinesis :

No media found
Verify that you are streaming to eu-central-1 and the fragment timecodes are correct, or download the Kinesis Video Streams Producer SDK to set up your device.

[jdelapla]

The logs show that you're dropping with flag 1536 very often. Can you check the dts and pts of the frames?

amazon-kinesis-video-streams-producer-sdk-cpp/src/gstreamer/gstkvssink.cpp

Line 1290 in 63537aa

LOG_DEBUG("Dropping frame with flag: " << GST_BUFFER_FLAGS(buf) << " for " << kvssink->stream_name);

Add the pts and dts to this log line, we'll make the change in an upcoming release as well.

[floatingpurr]

I build the project from e99e7e9. This is the corresponding section of the file you mentioned at that commit:

amazon-kinesis-video-streams-producer-sdk-cpp/src/gstreamer/gstkvssink.cpp

Lines 1101 to 1104 in e99e7e9

	if (isDroppable) {
	LOG_DEBUG("Dropping frame with flag: " << GST_BUFFER_FLAGS(buf));
	goto CleanUp;
	}

Do you mean something like this?

if (isDroppable) {
    LOG_DEBUG("Dropping frame with flag: " << GST_BUFFER_FLAGS(buf) << ", PTS: " << buf->pts << ", DTS: " << buf->dts << " for " << kvssink->stream_name);
    goto CleanUp;
}

Sorry if my question is trivial.

[jdelapla]

Yes, this is what I was requesting. Please generate logs with this change and see if it's the PTS or DTS being invalid that is causing the dropped frames.

Edit: I saw your logs were already posted in the next comment, please review my suggestion.

[floatingpurr]

Ok, I have just run the command with the customized build with info about pts and dts for the camera foo.

gst-launch-1.0 -v rtspsrc protocols=tcp location="rtsp://foo" short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name="YourStreamName" storage-size=128 access-key="foo-credential" secret-key="foo" aws-region="eu-central-1"

Here is the complete log: foo_tcp_pts_dts.log

It looks like no frame is put into the Kinesis stream.

[jdelapla]

Okay, so I went through the logs and as suspected it is a timestamp corruption issue.

2023-05-09 09:18:27 [140627329136192] DEBUG - Dropping frame with flag: 1536, PTS: 208075878, DTS: 18446744073709551615 for YourStreamName

The DTS value is a max int value. If this is not correctable by the source then you can edit this line in the kvssink:

amazon-kinesis-video-streams-producer-sdk-cpp/src/gstreamer/gstkvssink.cpp

Line 1288 in 63537aa

(GST_BUFFER_FLAG_IS_SET(buf, GST_BUFFER_FLAG_HEADER) && (!GST_BUFFER_PTS_IS_VALID(buf) || !GST_BUFFER_DTS_IS_VALID(buf)));

And remove the check on the DTS.

so this line should now be something like:

                        (GST_BUFFER_FLAG_IS_SET(buf, GST_BUFFER_FLAG_HEADER) && (!GST_BUFFER_PTS_IS_VALID(buf));

Let me know if this works.

[floatingpurr]

... aaand it works! Thank you very much for your kind help!

PS: I'm not an expert of RTSP or video streaming, what does the relaxing of the timestamp check imply?

PPS: any chance of having this option backed into the kvssink options (e.g,. check-dts)?

[jdelapla]

Yeah I'm planning to add the option to disable this in the next release. The reason this behavior is in place by default is because decoding timestamps are used internally to determine the sequential ordering of frames in the event that frames were delivered out of order for any reason.

If a stream has been accepting valid DTS values and then received a MAX_INT value like the DTS in your stream, then it would end up assuming all subsequent frames were invalid.

However in your case ALL the DTS values are invalid. There is actually a nice substitution in the kvssink a few lines below this droppable check that allows for this use case to behave normally. However if for whatever reason your RTSP was to suddenly start giving DTS values that are valid, then your stream would stop working.

TL;DR, either every DTS needs to be valid, or every DTS needs to be invalid. Otherwise the sink won't work. The implication of removing this check is the assumption that all frames will have an invalid DTS, and all frames will be sent to kvssink in order.