Start/Stop recording REST API doesn't work intermittently

[Mohit-3196]

Short description

When trying to start/stop recording for a stream using REST APIs, there are times when it doesn't work and gives success:false
(success=false, message=mp4 recording couldn't be started, dataId=null, errorId=0)
(success=false, message=mp4 recording couldn't be stopped, dataId=null, errorId=0)

It works most of the time and the issue is random.

Environment

• Ant Media Server version: Enterprise Edition 2.5.3

Steps to reproduce

1. Publish a stream to the server
2. Start/stop recording for the stream using the REST API call

Expected behavior

Start/Stop API calls should always work and gives success=true

Actual behavior

Start/Stop API call fails at times and gives success=false

Logs

https://drive.google.com/file/d/1yYWUv8UwaYmtfJ05a5FXv5OrY3mwD96w/view?usp=share_link
streamId=AV11-CAM

Join community.antmedia.io and be a part of Ant Media Server Community.

[Mohit-3196]

Additional Information:
Customer is publishing using a gstreamer pipeline and webrtcbin with a modified version of below script.
https://github.com/socieboy/webrtc-ant-media/blob/master/webrtc.py

There is no adaptive setting on the server.

2023-03-07 05:54:36,517 [network_thread - 42167] INFO i.a.enterprise.webrtc.WebRTCClient - Starting streaming for AV11-CAM
2023-03-07 05:54:36,517 [network_thread - 42167] INFO i.a.enterprise.webrtc.WebRTCClient - IceConnectionState: COMPLETED instance: 537440540
2023-03-07 05:54:36,517 [vert.x-eventloop-thread-0] INFO i.a.enterprise.webrtc.WebRTCClient - registering webrtc client 537440540 with codec H264 to webrtc adaptor for streamId AV11-CAM
2023-03-07 05:54:36,517 [vert.x-eventloop-thread-0] INFO i.a.enterprise.webrtc.WebRTCMuxer - register webrtc client(537440540) to webrtc muxer(480p - codec:H264) for stream: AV11-CAM current viewer count:1
2023-03-07 05:54:36,518 [network_thread - 42167] INFO org.webrtc.Logging - Tag:WebRtcAudioRecordExternal - Message:startRecording
2023-03-07 05:54:36,518 [network_thread - 42167] INFO i.a.enterprise.webrtc.WebRTCClient - DataChannel State Change for stream Id AV11-CAM
2023-03-07 05:54:36,518 [vert.x-eventloop-thread-0] INFO i.a.e.w.WebRTCPlayStreamInfoListener - A new viewer started streaming of AV11-CAM
2023-03-07 05:54:36,533 [network_thread - 42157] INFO i.a.enterprise.webrtc.WebRTCClient - IceConnectionState: CONNECTED instance: 1744757106
2023-03-07 05:54:36,533 [network_thread - 42157] INFO i.a.enterprise.webrtc.WebRTCClient - Timing:OnIceConnection Connected state takes 865ms hash:1744757106 remoteDescriptionSet:true
2023-03-07 05:54:36,533 [network_thread - 42157] INFO i.a.enterprise.webrtc.WebRTCClient - Starting streaming for AV11-CAM
2023-03-07 05:54:36,533 [network_thread - 42157] INFO i.a.enterprise.webrtc.WebRTCClient - IceConnectionState: COMPLETED instance: 1744757106
2023-03-07 05:54:36,533 [vert.x-eventloop-thread-16] INFO i.a.enterprise.webrtc.WebRTCClient - registering webrtc client 1744757106 with codec H264 to webrtc adaptor for streamId AV11-CAM
2023-03-07 05:54:36,533 [vert.x-eventloop-thread-16] INFO i.a.enterprise.webrtc.WebRTCMuxer - register webrtc client(1744757106) to webrtc muxer(480p - codec:H264) for stream: AV11-CAM current viewer count:2
2023-03-07 05:54:36,533 [network_thread - 42157] INFO org.webrtc.Logging - Tag:WebRtcAudioRecordExternal - Message:startRecording
2023-03-07 05:54:36,533 [network_thread - 42157] INFO i.a.enterprise.webrtc.WebRTCClient - DataChannel State Change for stream Id AV11-CAM
2023-03-07 05:54:36,533 [vert.x-eventloop-thread-16] INFO i.a.e.w.WebRTCPlayStreamInfoListener - A new viewer started streaming of AV11-CAM
2023-03-07 05:54:37,107 [https-jsse-nio2-0.0.0.0-5443-exec-10] INFO io.antmedia.rest.RestServiceBase - Recording method is called for AV11-CAM to make it true and record Type: mp4
2023-03-07 05:54:37,107 [https-jsse-nio2-0.0.0.0-5443-exec-10] WARN i.a.e.adaptive.EncoderAdaptor - There is no stream adaptor for mp4 recording for stream:AV11-CAM
2023-03-07 05:54:37,107 [https-jsse-nio2-0.0.0.0-5443-exec-10] WARN io.antmedia.rest.RestServiceBase - mp4 recording could not be started for stream: AV11-CAM
2023-03-07 05:54:37,574 [network_thread - 42143] INFO org.webrtc.Logging - Tag:WebRtcAudioTrackExternal - Message:GetPlayoutUnderrunCount
2023-03-07 05:54:37,574 [network_thread - 42143] INFO i.a.e.adaptive.WebRTCEncoderAdaptor - Publish Stats StreamId: AV11-CAM TransferedByte: 362309 Stream Duration: 3851 Bitrate: 94 Source IP: User-Agent: [Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36]
2023-03-07 05:54:39,098 [https-jsse-nio2-0.0.0.0-5443-exec-1] INFO io.antmedia.rest.RestServiceBase - Recording method is called for AV11-CAM to make it true and record Type: mp4
2023-03-07 05:54:39,098 [https-jsse-nio2-0.0.0.0-5443-exec-1] WARN i.a.e.adaptive.EncoderAdaptor - There is no stream adaptor for mp4 recording for stream:AV11-CAM
2023-03-07 05:54:39,098 [https-jsse-nio2-0.0.0.0-5443-exec-1] WARN io.antmedia.rest.RestServiceBase - mp4 recording could not be started for stream: AV11-CAM
2023-03-07 05:54:41,520 [vert.x-eventloop-thread-5] INFO i.a.enterprise.webrtc.WebRTCAdaptor - Client:537440540 for stream AV11-CAM current video bitrate: 991973 audio bitrate: 96000 webrtc client target bitrate: 5016000
2023-03-07 05:54:41,534 [vert.x-eventloop-thread-19] INFO i.a.enterprise.webrtc.WebRTCAdaptor - Client:1744757106 for stream AV11-CAM current video bitrate: 987036 audio bitrate: 96000 webrtc client target bitrate: 4997000
2023-03-07 05:54:42,569 [network_thread - 42143] INFO org.webrtc.Logging - Tag:WebRtcAudioTrackExternal - Message:GetPlayoutUnderrunCount
2023-03-07 05:54:42,570 [network_thread - 42143] INFO i.a.e.adaptive.WebRTCEncoderAdaptor - Publish Stats StreamId: AV11-CAM TransferedByte: 927872 Stream Duration: 8847 Bitrate: 104 Source IP: User-Agent: [Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36]
2023-03-07 05:54:43,973 [ForkJoinPool.commonPool-worker-13] INFO i.a.e.w.codec.VirtualVideoEncoder - frame id: 2575447339 capture TimeNs: 10220000000

[ashishkumar-07]

This issue can be reproduces from UI also.
PSB snapshots.

[sumitsanketinfinite]

Hi Team,

This issue has been going on for more than 1 month and we still have no resolution.
I am sorry but we are really getting impression that this issue is not being given correct priority. Recording of stream is core feature of our product which has not been working because of which our product has become really unusable. Unfortunately ant-media claims that this is basic feature they provide but it is not working.

Please assign a definite ETA to resolving the issue. If needed please involve higher level escalation matrix to the issue because it seems that right set of people are not involved or right priority is not assigned.

[mekya]

Hi @sumitsanketinfinite,

Thank you for raising your concern. Let me discuss this issue with my colleagues in today's meeting.

Regards
Oguz

[Mohit-3196]

Hi @sumitsanketinfinite ,
Thank you again for your comments here. I think we had this conversation on the ticket thread so let me update here as well.

We understand that the issue you are experiencing with the recording feature has been ongoing for more than a month, I have been involved with you all since the start and will continue to be till we reach a solution, and I apologize for any inconvenience this has caused you.

The team is highly occupied with some very high-priority development tasks and we've commitments to the paid support users.

Please know that we take this matter seriously and we are doing everything to resolve it as quickly as possible. We appreciate your feedback and want to assure you that we have assigned the required priority to this issue. You can make the priority assumption based on the fact that not just me but another colleague of mine as well as one of the lead developers have been involved in this case and have had several calls with you guys to be in sync and working with you to reproduce the problem and identify a solution, but as you have mentioned this issue happens in a specific environment and specifically only you have reported this issue, this has proven to be challenging.

We are committed to delivering a high-quality product that meets your needs, and we understand that the recording feature is critical to your use case. We want to assure you that we will continue to work on this until the issue is resolved, and we will keep you updated on our progress.

At this time, we do not have a definite ETA for resolving the issue, but we are making every effort to provide you with regular updates as we make progress. We appreciate your patience and understanding as we work to resolve this issue.

As you know, last week we shared a dedicated build with you to test and reproduce the issue. Based on those tests, we reviewed the log files but unfortunately, there weren't any conclusive entries to understand the cause of the issue.

To move forward, I have shared another build with your team last evening and I request you to make tests with that again to reproduce the issue. Once it is reproduced, we will review the logs again and hopefully, we can get into the root cause.

Once again, I appreciate your patience and am pretty hopeful that we are nearing finalizing the issue :)

Thank you,
Mohit Dubey

[Mohit-3196]

This issue has been resolved and the recording works fine now.
There is another issue with the recording size for which the investigation is under process.

[danavramescu]

@Mohit-3196 , is the 'recording size' issue you're investigating still producing issues like the ones described above? we're still having them, even today. Enterprise Edition 2.6.4 20230827_1351

Logs:
2023-10-19 06:35:56,250 [https-jsse-nio2-0.0.0.0-5443-exec-181] INFO io.antmedia.rest.RestServiceBase - Recording method is called for 11068081 to make it true and record Type: mp4 resolution:0
2023-10-19 06:35:56,250 [https-jsse-nio2-0.0.0.0-5443-exec-181] WARN i.a.e.adaptive.EncoderAdaptor - There is no stream adaptor for mp4 recording for stream:11068081
2023-10-19 06:35:56,250 [https-jsse-nio2-0.0.0.0-5443-exec-181] WARN io.antmedia.rest.RestServiceBase - mp4 recording could not be started for stream: 11068081

2023-10-19 06:35:57,223 [https-jsse-nio2-0.0.0.0-5443-exec-131] INFO io.antmedia.rest.RestServiceBase - Recording method is called for 11068081 to make it true and record Type: mp4 resolution:0
2023-10-19 06:35:57,223 [https-jsse-nio2-0.0.0.0-5443-exec-131] WARN i.a.e.adaptive.EncoderAdaptor - There is no stream adaptor for mp4 recording for stream:11068081
2023-10-19 06:35:57,223 [https-jsse-nio2-0.0.0.0-5443-exec-131] WARN io.antmedia.rest.RestServiceBase - mp4 recording could not be started for stream: 11068081

2023-10-19 06:03:18,640 [https-jsse-nio2-0.0.0.0-5443-exec-115] INFO io.antmedia.rest.RestServiceBase - Recording method is called for 11741491 to make it true and record Type: mp4 resolution:0
2023-10-19 06:03:18,640 [https-jsse-nio2-0.0.0.0-5443-exec-115] WARN i.a.e.adaptive.EncoderAdaptor - There is no stream adaptor for mp4 recording for stream:11741491
2023-10-19 06:03:18,640 [https-jsse-nio2-0.0.0.0-5443-exec-115] WARN io.antmedia.rest.RestServiceBase - mp4 recording could not be started for stream: 11741491

This affects different broadcasters, but not all of them.
TO NOTE: for these broadcasters, the recording cannot be started at all, at any point during the livestream, always returning the error "mp4 recording couldn't be started". In the past this issue was solved by running a service restart, after which we could start recordings again for these broadcasters, but we cannot do that at the moment until your colleagues investigate another issue we have an open ticket for.

---

For reference, will also show a logs snippet of an "OK" recording-start:

2023-10-19 07:03:40,381 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO io.antmedia.rest.RestServiceBase - Recording method is called for 10676237 to make it true and record Type: mp4 resolution:0
2023-10-19 07:03:40,382 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO i.a.e.adaptive.EncoderAdaptor - Recording video codec is = io.antmedia.enterprise.adaptive.video.SFUForwarder@1f911725 for streamId:10676237
2023-10-19 07:03:40,383 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO io.antmedia.muxer.Mp4Muxer - Date time resource name: 10676237-2023-10-19_07-03-40.383 local date time: 2023-10-19_07-03-40.383
2023-10-19 07:03:40,384 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO i.a.e.adaptive.EncoderAdaptor - start recording video stream index: 1 audio stream index:0
2023-10-19 07:03:40,384 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO i.a.e.adaptive.EncoderAdaptor - Muxer:Mp4Muxer is adding to Encoder:AACEncoder for stream:10676237
2023-10-19 07:03:40,385 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO io.antmedia.muxer.Mp4Muxer - Adding timebase to the input time base map index:0 value: 1/44100 for stream:10676237
2023-10-19 07:03:40,385 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO i.a.e.adaptive.EncoderAdaptor - Muxer:Mp4Muxer is adding to Encoder:SFUForwarder for stream:10676237
2023-10-19 07:03:40,386 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO io.antmedia.muxer.Mp4Muxer - Header is written for stream:10676237 and url:/usr/local/antmedia/webapps/WebRTCAppEE/streams/10676237-2023-10-19_07-03-40.383.mp4.tmp_extension
2023-10-19 07:03:40,386 [https-jsse-nio2-0.0.0.0-5443-exec-129] INFO i.a.e.adaptive.EncoderAdaptor - Start mp4 recording for stream: 10676237
2023-10-19 07:03:40,386 [https-jsse-nio2-0.0.0.0-5443-exec-129] WARN io.antmedia.rest.RestServiceBase - mp4 recording is started for stream: 1067623