random video interrupts

[JRafiei]

Hi jlaine!
Sometimes, when we are video streaming, the frames stops. we track the problem to the rtcrtpreceiver.py file and it seems some frames are None and so they don't inserted in the queue. Do you have any ideas why this happens and what should we do to solve the problem? Is it a bug or an expected behavior?
Here is the related section in rtcrtpreceiver.py module for convenience:

async def _handle_rtp_packet(self, packet):
     self.__log_debug('< %s', packet)
     if packet.payload_type in self._decoders:
         decoder = self._decoders[packet.payload_type]
         loop = asyncio.get_event_loop()

         # RTCP
         if self.__remote_ssrc is None:
             self.__remote_ssrc = packet.ssrc
             self.__remote_counter = LossCounter(packet.sequence_number)
         else:
             self.__remote_counter.add(packet.sequence_number)

         if self._kind == 'audio':
             # FIXME: audio should use the jitter buffer!
             audio_frame = await loop.run_in_executor(None, decoder.decode, packet.payload)
             await self._track._queue.put(audio_frame)
         else:
             # check if we have a complete video frame
             self._jitter_buffer.add(packet.payload, packet.sequence_number, packet.timestamp)
             payloads = []
             got_frame = False
             last_timestamp = None
             for count in range(self._jitter_buffer.capacity):
                 frame = self._jitter_buffer.peek(count)
                 if frame is None:
                     break
                 if last_timestamp is None:
                     last_timestamp = frame.timestamp
                 elif frame.timestamp != last_timestamp:
                     got_frame = True
                     break
                 payloads.append(frame.payload)

             if got_frame:
                 self._jitter_buffer.remove(count)
                 video_frames = await loop.run_in_executor(None, decoder.decode, payloads)
                 for video_frame in video_frames:
                     await self._track._queue.put(video_frame)

I think the important section is the last else block, where you are putting the video_frames to the queue. It creates the video frames only when got_frame is True and other frames get discarded.
Our clients are android smart phones and we are testing in a local network.

Based on another issue we changed the buffer capacity from 32 to 64 and we got slightly better results. Do you think the buffer capacity is an important factor here?

[jlaine]

Looks like it might be a duplicate of #26

[JRafiei]

Yeah you are right. Actually I was reffering to this issue in the first place. But I can't understand the rationale behind it. Is it safe to apply those changes? By the way could you direct me to the neccesary informations about creating video frames from the incoming packets?

[jlaine]

If you re-read the code, I don't believe we are "discarding" any frames. We:

• run over the chunks present in the jitter buffer to determine if we have a complete frame (one video frame may span several RTP packet), and if so we set got_frame to True

• if we did receive a complete frame, we remove it from the jitter buffer and pass it to the codec for decoding

• otherwise we need to wait for the missing packets to arrive

What kind of resolution are you using? If you are having issues with the jitter buffer capacity I'm guessing you have very large images. One interesting metric would be to print count in the if got_frame branch.

[bl1nder]

It seems i have something similar . Based on your server example I'm streaming video from browser to server. on low resolution it works fine. After setup constraints to 1280 x 720 , framerate:5. After received about 50 frames server stuck. Server host machine profiling with tcpdump shows packets which still coming to server.
See code below

 @pc.on('track')
    def on_track(track):
        if track.kind == 'video':            
            pc._consumers.append(asyncio.ensure_future(consume_video(track)))

async def consume_video(track):
    
    Drain incoming video, and echo it back.
    logging.error('consume_video')
    count = 0
    while True:

        logging.error('start loop')
        frame_remote = await track.recv()
        count += 1
        logging.error(count)
        logging.error(datetime.datetime.now().time())

Could you help me with this?(add some thoughts any help) I'm beginners with python. Thanks in advance

[jlaine]

Could you please try running the example in verbose mode (python examples/server/server.py -v), and paste some of the output (especially towards the end)?

[jlaine]

@bl1nder I believe that's a different issue: I noticed that when you request a 1280x720 resolution from getUserMedia, Chrome will initially send the VP8 video out at a lower resolution (for instance 640x360), then progressively increase the resolution. Decoding works fine, but echoing these frames back failed when the resolution changed as the encoder needs to be torn down / recreated.

That should be fixed by ad0dadf

[JRafiei]

Thanks jlane for explaing the code and sorry for late response. I checked the parameters that you mentioned and here are the results:
I printed the Video frames size and they are 480x640. (portrait)
The count is 1 or 2 most of the times but occasionally it becomes higher values like 15 or 19.
By the way we changed the buffer capacity to 128 and MAX_DROPOUT to 500 to reduce the freeze time. And it seems we are getting better results. Do you think these are resonable changes?

[jlaine]

Could you please just try the latest code from git? It uses a buffer capacity of 128, and MAX_DROPOUT is gone.

[JRafiei]

Yes, sure!
After testing with the latest version, the video quality has increased significantly and it is more stable. Sometimes the video get pixelated and stopped for a short time but it resumed to the normal state after that. Thanks a lot for your efforts.
during testing the video call we get two new errors that may be informative.

Task exception was never retrieved
future: <Task finished coro=<RTCDtlsTransport.__run() done, defined at /usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py:365> exception=Error('replay check failed (bad index)',)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py", line 368, in __run
await self._recv_next()
File "/usr/local/lib/python3.6/site-packages/aiortc-0.9.1-py3.6-linux-x86_64.egg/aiortc/rtcdtlstransport.py", line 427, in _recv_next
data = self._rx_srtp.unprotect(data)
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 178, in unprotect
return self.__process(packet, lib.srtp_unprotect)
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 198, in __process
_srtp_assert(func(self._srtp[0], self._cdata, len_p))
File "/usr/local/lib/python3.6/site-packages/pylibsrtp/__init__.py", line 52, in _srtp_assert
raise Error(ERRORS[rc])
pylibsrtp.Error: replay check failed (bad index)

and

Exception in callback Transaction.__retry()
handle: <TimerHandle when=1566464.094515949 Transaction.__retry()>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/asyncio/events.py", line 127, in _run
self._callback(*self._args)
File "/usr/local/lib/python3.6/site-packages/aioice-0.6.2-py3.6.egg/aioice/stun.py", line 257, in __retry
self.__future.set_exception(exceptions.TransactionTimeout())
asyncio.base_futures.InvalidStateError: invalid state

After these errors, video playback completely stops and the ice connection become disconnected and then failed.

[bl1nder]

Hi i had the same error as @JRafiei mentioned. @JRafiei Did you handle this type replay check failed (bad index) of error? Did add some reconnect code for this case? it's hard to reproduce. Now it work fine for me, but i'm not sure on 100% this sutiation will not happen again. Please share any info about this. thanks in advance.

[jlaine]

Ok can we please open separate tickets to track the new issues?