SRT H265 Streams - Stops and Becomes Unplayable After Long Viewing Session #5504

timantmedia · 2023-08-25T08:57:58Z

Short description

SRT H265 streams are streamed to Ant Media Server and then viewed in a browser.

If we start viewing the streams for a long playback session, after some point in time the stream goes black, and there is the ability to view the video in the Ant Media Server portal player or directly in our application in the browser.

The amount of time before this occurs does not appear to be fixed, and is usually measured in hours. The server is not under any significant strain with values at around 30-40%.

Also experienced seeing it stream with the broadcasting status offline yet they are in fact playable and broadcasting.

Environment

Operating system and version: Azure VM F8 instance
Java version:
Ant Media Server version: Enterprise Edition 2.6.2
Browser name and version:
VP8 codec with 360p ABR

Steps to reproduce

Select stream for viewing
Leave the stream playing for a long period of 8+ hours
Stream becomes unplayable and refreshing the browser or using a different machine does not resolve the issue
Other streams on Ant Media Server still remain functioning
The only known solution for fixing playback is to restart Ant Media Service
The stream is still in broadcast status yet not playback/black screen

The stream is currently being sent to srt://test.antmedia.io:4200?streamid=FD103288/GRzNI6RxfaRx8R1o1692779054559 and is playing fine. It's been live for only a short time as of the creation of this issue.

Update: the live stream is now displaying the expected symptoms and is not playing any longer

Access to customer test server is in the ticket. This is similar issues that do not seem to have been resolved:

#4434

I tried forwarding the stream to an RTMP endpoint on OVH. The customers test server is showing the error:

2023-09-06 13:27:00,565 [vert.x-worker-thread-90] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,565 [vert.x-worker-thread-90] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,565 [vert.x-eventloop-thread-2] INFO io.antmedia.muxer.MuxAdaptor - prepareMuxer for stream:jbp3cP0E9CxfSFDC1693850677302 muxer:RtmpMuxer 2023-09-06 13:27:00,565 [vert.x-eventloop-thread-2] INFO io.antmedia.muxer.RtmpMuxer - Creating outputFormatContext 2023-09-06 13:27:00,565 [vert.x-eventloop-thread-2] WARN io.antmedia.muxer.RtmpMuxer - Stream is not added for muxing to jbp3cP0E9CxfSFDC1693850677302_480p for stream:jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,565 [vert.x-eventloop-thread-2] WARN io.antmedia.muxer.MuxAdaptor - muxer add streams returns false flv 2023-09-06 13:27:00,565 [vert.x-eventloop-thread-2] ERROR io.antmedia.muxer.MuxAdaptor - RTMP prepare returned false so that rtmp pushing to rtmp://ovh36.antmedia.io/H265/K1ADNRSFCM62HKL81694006707388 for jbp3cP0E9CxfSFDC1693850677302 didn't started 2023-09-06 13:27:00,566 [vert.x-eventloop-thread-2] INFO i.a.e.adaptive.EncoderAdaptor - Muxer:RtmpMuxer is adding to Encoder:SFUForwarder for stream:jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,566 [vert.x-eventloop-thread-2] INFO io.antmedia.muxer.RtmpMuxer - Creating outputFormatContext 2023-09-06 13:27:00,566 [vert.x-eventloop-thread-2] INFO io.antmedia.muxer.RtmpMuxer - Adding video stream index:0 for stream:rtmp://ovh36.antmedia.io/H265/K1ADNRSFCM62HKL81694006707388 2023-09-06 13:27:00,614 [vert.x-worker-thread-111] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,614 [vert.x-worker-thread-111] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,614 [vert.x-worker-thread-111] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,614 [vert.x-worker-thread-111] WARN io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: jbp3cP0E9CxfSFDC1693850677302 2023-09-06 13:27:00,615 [vert.x-worker-thread-111] WARN io.antmedia.muxer.RtmpMuxer - Not writing to RTMP muxer because it's not started for rtmp://ovh36.antmedia.io/H265/K1ADNRSFCM62HKL81694006707388 2023-09-06 13:27:00,620 [Thread-7] WARN i.a.enterprise.srt.SRTAdaptor - SRT stream map does not match any stream for socket:0

The OVH server is showing the error

2023-09-06 14:15:28,209 [vert.x-worker-thread-117] ERROR io.antmedia.muxer.MuxAdaptor - There is no video and audio in the incoming stream: 0eID2OyBFv1I4idx1694004694538 closing rtmp connection

Expected behavior

Should play without issue

Actual behavior

After a few hours, the stream goes blank, and is not able to play. The only way to resolve is to restart the server.

FD103288

Logs

Place logs on pastebin or elsewhere and put links here

Ask your questions on Ant Media Github Discussions

The text was updated successfully, but these errors were encountered:

timantmedia · 2023-09-07T07:48:17Z

@mekya Customer is looking for alternatives because the issue is a blocker. They are looking to scale AMS to 100+ broadcasters if the issue is resolved.

mekya · 2023-09-07T08:05:19Z

Hi @timantmedia,

As far as I see you've reproduced the problem in our test server. Could you please let us have the logs on the test server and let us know the stream id?

Regards
Oguz

timantmedia · 2023-09-07T08:11:14Z

@mekya here is the ticket with logs and stream id https://antmedia.freshdesk.com/a/tickets/106818 for stream id GRzNI6RxfaRx8R1o1692779054559

timantmedia · 2023-09-07T08:17:57Z

@mekya also customer test server logs are here https://antmedia.freshdesk.com/a/tickets/106777 for stream id jbp3cP0E9CxfSFDC1693850677302

timantmedia · 2023-09-11T09:44:16Z

I reproduced the SRT stream getting stuck in broadcast state by doing the following:

Running this ffmpeg command

ffmpeg -re -i example.webm -vcodec libvpx -f mpegts "srt://ant-media-server:4200?streamid=LiveApp/stream9"

The file exists, but I am trying to send vp8 stream to AMS. The terminal shows it's sending data, but once it's finished, the live stream appears in the web panel as broadcasting but nothing is actually getting live streamed and its stuck in broadcasting state.

timantmedia · 2023-09-15T08:52:47Z

@mekya Just wondering if there was any update I can provide the customer?

mekya · 2023-09-21T08:14:32Z

@mekya here is the ticket with logs and stream id https://antmedia.freshdesk.com/a/tickets/106818 for stream id GRzNI6RxfaRx8R1o1692779054559

@timantmedia
I've analyzed the logs. I see there are some other things in the server as a plugin. For instance, it gives this error
->
java.lang.NoClassDefFoundError: Could not initialize class io.antmedia.app.NativeInterface$JNA_RTSP_SERVER

I also see that there are some crashes which may be related to the plugin as well. I think it's good to understand how the user plays the stream and what other plugins/components are in the server

In the perspective of fixing the curent problem,

Callback is called hsversion: 5  stream id:  FD103288/GRzNI6RxfaRx8R1o1692779054559
2023-09-03 01:19:45,596 [JavaCPP Thread ID 8477] ERROR i.a.enterprise.srt.SRTAdaptor - Sending stream id in use error for stream:org.bytedeco.javacpp.BytePointer[address=0x7f874affbc20,position=0,limit=0,capacity=0,deallocator=null] scope:FD103288

According to the logs above, It seems that the stream is stucked at broadcasting state and it does not accept the stream It always returning streamIdInUse error as in the logs I think we can fix it easily. This is why user don't play it.
Additionally, It will be very helpful if we know the older logs to understand how the stream stucks at the broadcasting state to confirm the solution we provide is permanent.

mekya · 2023-09-21T08:25:58Z

@mekya also customer test server logs are here https://antmedia.freshdesk.com/a/tickets/106777 for stream id jbp3cP0E9CxfSFDC1693850677302

@timantmedia , these logs does not help me a lot understand the problem because it's full of

WARN  io.antmedia.muxer.RtmpMuxer - Header is not written yet for writing video packet for stream: AZbJMpk7n5OJsRwx1693862978693

mekya · 2023-09-21T08:27:35Z

I reproduced the SRT stream getting stuck in broadcast state by doing the following:

Running this ffmpeg command

ffmpeg -re -i example.webm -vcodec libvpx -f mpegts "srt://ant-media-server:4200?streamid=LiveApp/stream9"

The file exists, but I am trying to send vp8 stream to AMS. The terminal shows it's sending data, but once it's finished, the live stream appears in the web panel as broadcasting but nothing is actually getting live streamed and its stuck in broadcasting state.

I think VP8 is not compatible for mpegts. No matter, server should not behave like this. Let me check.

timantmedia · 2023-09-21T11:17:21Z

@mekya So he is using a Makito Encoder and it is live camera content. Do you need an exact model? Or other details? It's sending SRT H265.

I can ask if he has older logs, but these we generated from the support page of the web panel. So does that only send a portion of logs from the server?

I'll ask if he is using any Ant Media Server plugins or do you mean some server plugins or something else?

mekya · 2023-09-24T11:07:53Z

Currently, I've reproduced some scenarios in the logs and implemented the fixes and test codes. It's running in CI right now.

The issue below is fixed.

Callback is called hsversion: 5  stream id:  FD103288/GRzNI6RxfaRx8R1o1692779054559
2023-09-03 01:19:45,596 [JavaCPP Thread ID 8477] ERROR i.a.enterprise.srt.SRTAdaptor - Sending stream id in use error for stream:org.bytedeco.javacpp.BytePointer[address=0x7f874affbc20,position=0,limit=0,capacity=0,deallocator=null] scope:FD103288

The problem you mentioned is fixed. I mean vp8 is not compatible with mpegts and we stop the ingestion after it's timeout. So that it does not enter the blocking state

I reproduced the SRT stream getting stuck in broadcast state by doing the following:

Running this ffmpeg command

ffmpeg -re -i example.webm -vcodec libvpx -f mpegts "srt://ant-media-server:4200?streamid=LiveApp/stream9"

After CI passes and snapshot is ready on website, we can deliver the user to confirm if it's working for them.

Happy Sunday Coding :)
Cheers

timantmedia · 2023-09-25T10:26:13Z

@mekya there was a snapshot version generated this morning. I assume this contains the fixes?

mekya · 2023-09-25T11:16:52Z

@mekya there was a snapshot version generated this morning. I assume this contains the fixes?

Yes

timantmedia · 2023-10-09T09:04:41Z

@mekya The fix did not resolve the customer's issue. The issue is still present, the video says broadcasting, and the encoder is still outputting data, but the video is unplayable, despite using the latest snapshot version.

mekya · 2023-10-09T10:59:54Z

Then we need to get the logs again from the ant media server and streamId that they encounter this problem.

timantmedia · 2023-10-09T15:10:09Z

Logs are attached, I was waiting for the customer to confirm the stream id.

ant-media-server.log (3) 2.zip

timantmedia · 2023-10-18T08:22:40Z

@mekya The customer is not responding to confirm which stream id is showing the issue, however, this is a list of the stream IDs I found in the log.

z99sXMsjIf4npeot1696355403015
Q4n7GppBJ4wz7AHC1696352601320
78QlHP1D1iaLyKxF1696352615354
vcISTmzOaEd6vre21696352610835
2MkpBJqJUpNhVSxK1696352622571

In case it's possible to check the logs on just these few streams to find anything unusual. I asked the customer to comment on this issue.

timantmedia · 2023-11-12T21:08:48Z

Taken out of last sprint, can it be scheduled for this week @mekya

mekya · 2023-11-13T08:12:54Z

Hi @timantmedia
If there is some space to get some new issues, yes we can schedule it again.

Regards
Oguz

timantmedia added awaiting-confirmation Awaiting confirmation from user to close the issue. investigating labels Aug 25, 2023

timantmedia added bug Confirmed investigating awaiting-confirmation Awaiting confirmation from user to close the issue. Priority: Urgent and removed investigating awaiting-confirmation Awaiting confirmation from user to close the issue. bug Confirmed labels Sep 6, 2023

mekya mentioned this issue Sep 7, 2023

Hanging SRT streams published from Makito X4 SDI Encoder #4434

Closed

mustafaboleken mentioned this issue Sep 11, 2023

Random crash with SRT #5472

Open

mekya self-assigned this Sep 11, 2023

mekya mentioned this issue Sep 24, 2023

Fix srt broadcast stuck #5606

Merged

mekya assigned timantmedia Sep 25, 2023

timantmedia mentioned this issue Nov 16, 2023

SRT Ingest does not reconnect after connection loss #5736

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SRT H265 Streams - Stops and Becomes Unplayable After Long Viewing Session #5504

SRT H265 Streams - Stops and Becomes Unplayable After Long Viewing Session #5504

timantmedia commented Aug 25, 2023 •

edited

timantmedia commented Sep 7, 2023

mekya commented Sep 7, 2023

timantmedia commented Sep 7, 2023 •

edited

timantmedia commented Sep 7, 2023

timantmedia commented Sep 11, 2023

timantmedia commented Sep 15, 2023

mekya commented Sep 21, 2023 •

edited

mekya commented Sep 21, 2023

mekya commented Sep 21, 2023

timantmedia commented Sep 21, 2023

mekya commented Sep 24, 2023

timantmedia commented Sep 25, 2023

mekya commented Sep 25, 2023

timantmedia commented Oct 9, 2023

mekya commented Oct 9, 2023

timantmedia commented Oct 9, 2023

timantmedia commented Oct 18, 2023

timantmedia commented Nov 12, 2023

mekya commented Nov 13, 2023

SRT H265 Streams - Stops and Becomes Unplayable After Long Viewing Session #5504

SRT H265 Streams - Stops and Becomes Unplayable After Long Viewing Session #5504

Comments

timantmedia commented Aug 25, 2023 • edited

Short description

Environment

Steps to reproduce

Expected behavior

Actual behavior

Logs

timantmedia commented Sep 7, 2023

mekya commented Sep 7, 2023

timantmedia commented Sep 7, 2023 • edited

timantmedia commented Sep 7, 2023

timantmedia commented Sep 11, 2023

timantmedia commented Sep 15, 2023

mekya commented Sep 21, 2023 • edited

In the perspective of fixing the curent problem,

mekya commented Sep 21, 2023

mekya commented Sep 21, 2023

timantmedia commented Sep 21, 2023

mekya commented Sep 24, 2023

timantmedia commented Sep 25, 2023

mekya commented Sep 25, 2023

timantmedia commented Oct 9, 2023

mekya commented Oct 9, 2023

timantmedia commented Oct 9, 2023

timantmedia commented Oct 18, 2023

timantmedia commented Nov 12, 2023

mekya commented Nov 13, 2023

timantmedia commented Aug 25, 2023 •

edited

timantmedia commented Sep 7, 2023 •

edited

mekya commented Sep 21, 2023 •

edited