Stream stuck in broadcasting state when one of the node cluster gets inaccessible

[USAMAWIZARD]

Short description

In a scenario where there are two nodes in the cluster and client is publishing RTMP stream to one of the node and that node gets inaccessible then the streams will be stuck in broadcasting mode.then we cannot delete that stream as well they are just stuck.

Environment

• Operating system and version:
• Java version:
• Ant Media Server version:2.6.0
• Browser name and version:

Steps to reproduce

1. two nodes in the cluster
2. streams publishing one of the node
3. make that node inaccessible (Crash the node )
4. streams will be stuck in broadcasting state.and cannot be deleted as well.

Expected behavior

streams should be offline

Actual behavior

streams stuck in broadcasting

When I check the broadcast object using REST API.
the origin IP of the broadcast is the node which was disconnected.

Logs

Place logs on pastebin or elsewhere and put links here

Ask your questions on Ant Media Github Discussions

[sahiljain2497]

This might be related as well. I am on Enterprise Edition 2.6.2 20230623_0843

Screen.Recording.2023-07-22.at.11.02.04.PM.mov

[lastpeony]

This could be connected to a web panel attempting to send a request to a node that lacks the stream. Addressing this by directing requests to the appropriate cluster nodes can resolve the issue. Further information will be provided shortly.

[lastpeony]

Please check mentioned PR for fix and bug explanation.

[SebastienGautier]

I updated to 2.6.3 but I still have 2 streams stuck in the broadcasting state that I am unable to stop or delete
Was this fix supposed to handle that or only avoid new streams being stuck ?

[mekya]

Hi @SebastienGautier,
Yes, this fix is supposed to handle your case as well. Is there a chance that you can let us take a look at your setup by sharing webpanel credentials and ssh if possible?

You can send it to support@antmedia.io

[SebastienGautier]

Hi @mekya
It's now our production instance, I can't give access to this one
The error I get in the logs when I tried to delete a bugged stream is the following:
2023-09-19 07:12:59,080 [http-nio2-0.0.0.0-5080-exec-9] ERROR io.antmedia.rest.RestServiceBase - Please send a Delete Broadcast request to the x.x.x.x node or Delete Broadcast in a stopped broadcast.

[mekya]

Hi @SebastienGautier ,
This is not supposed to happen. I understand that why you don't provide access.

So how shall we proceed? If you want to speed up the process, you can send an email to support@antmedia.io