Memory issues in broadcast upload extensions when using VP8

[yahmad]

Description

We are encountering memory issues in broadcast extensions when trying to create P2P rooms with:

• an iOS participant providing the video track through a broadcast extension
• a couple of participants just viewing the video track (all participants have preferred video codecs set to H.264)
• at least one participant subscribing to the video track on an Android device which does not support H.264 hardware support

We are already downscaling the video quality to reduce memory usage, and also downscaling dynamically based on system memory warnings, but when a participant joins a multi-person P2P room requiring a VP8 codec for video, the broadcast extension crashes fairly quickly due to exceeding memory limits.

We were hoping this was an edge case, but did some testing and found there are quite a few devices which don't have hardware support for H.264 (a few we personally tested are listed below).

Questions

1. Am I correct in assuming there is an encoder created per participant, even if multiple participant connections are using the same codecs?
2. Are there any plans/technical barriers to making use of shared encoders on a video track provider?
3. Is there anything you recommend we do to optimise memory usage in a P2P room?

Memory stats

It looks like each H.264 compatible participant added to the room increases memory usage by about 2.5MB (an encoder per participant?), but when a participant without this capability joins, it falls back to using VP8 which requires upto 15MB of memory to encode.

Android problem devices

Encountered devices which do not have H.264 hardware support (mid-level devices from manufacturers based in China/East-Asia seem to especially not provide H.264 hardware support):

• Leagoo Kiicaa Power
• Elephone P8 mini
• HomTom S8 (though the spec sheet suggests it does?)

Versions

Video iOS SDK

TwilioVideo v2.1 via CocoaPods

Xcode

Xcode v9.4.1

iOS Version

iOS 11.4

iOS Device

iPhone 7
iPhone X

[ceaglest]

Hi @yahmad,

Thanks for writing in. Let me consult with @piyushtank before writing a full answer to your inquiry.

In the meantime, we have discussed broadcast uploader extensions and some of the challenges in this issue: #7

Best,
Chris

[yahmad]

Thanks for writing in. Let me consult with @piyushtank before writing a full answer to your inquiry.

Thanks @ceaglest, look forward to hearing back.

In the meantime, we have discussed broadcast uploader extensions and some of the challenges in this issue: #7

I think we have got a working solution (or at least good enough within iOS limitations) for some of the problems mentioned in the other issue. I am hoping there might be some codec specific information / suggestions that might shed some light. 🤞

[piyushtank]

@yahmad, As you already figured, the memory grows beyond max limit. As mentioned as a known issue in our changelogs, 1280x720 resolution @ 30fps is not currently supported with H.264. By default, ReplayKit provides pixel buffers with native size of the screen. e.g 1080x1920 for iPhone 6s plus. If you downscale lower than 1280x720 @30fps video resolution, you should be able to broadcast H.264. Also, in my experience, if the video frame size is 640x480 the extension was using ~20MB of memory with VP8.

In future, we are planning to provide APIs where you just need to provide provide the pixel buffers and the video sdk will downscale it for you. Also, we are planning to add a sample app which demonstrates the ReplayKit extension usage with Video SDK in next few days. I will let you know when it becomes available.

Let me know if you have any questions.

[ceaglest]

Hey @yahmad,

Some feedback before I leave the office, to compliment what Piyush posted.

Am I correct in assuming there is an encoder created per participant, even if multiple participant connections are using the same codecs?

It depends on the type of Room you are in. In a Peer-to-Peer Room you encode a distinct stream for each Participant, but this is not the case in a Group Room.

Encoders Used
---------------
Peer-to-Peer = LocalVideoTracks * RemoteParticipants
Group = LocalVideoTracks
Group, VP8 Simulcast = LocalVideoTracks * 3

Are there any plans/technical barriers to making use of shared encoders on a video track provider?

It's not feasible to share a single encoder when different rates are being requested on different media connections. Ultimately, its a question of topology.

Is there anything you recommend we do to optimise memory usage in a P2P room?

Piyush made some good suggestions above about downscaling. We are going to publish sample code which demonstrates the best practices using our current SDK. After 2.3.0-preview1 we will consider augmenting our capture APIs so you can request that the encoder perform downscaling. Also, there are other ways we can reduce memory usage in the encoder/capture pipeline. This was discussed in a little more detail in the linked issue (#7).

Regards,
Chris

[yahmad]

we are planning to add a sample app which demonstrates the ReplayKit extension usage with Video SDK

Cheers @piyushtank , we are currently at least halving our resolution, but not sure about the specific resolution so I will find out if we can reduce it further.

In a Peer-to-Peer Room you encode a distinct stream for each Participant, but this is not the case in a Group Room.

@ceaglest Thanks, I figured this was how the encoders were probably working.

Having a sample app will be really useful - thanks for providing this and for responding so quickly!

[piyushtank]

We have an open which demonstrates ReplayKit broadcast service with TwilioVideo SDK - twilio/video-quickstart-ios#287
The PR demonstrates ScreenSharing while connected to a Room using ReplayKit broadcast service extension. Please note it is a work in progress PR and we plan to complete the TODOs items soon.
Let me know if you have any questions.

[yahmad]

Thanks, we will take a look!

[ceaglest]

Hello @yahmad (and other ReplayKit Developers),

We have released 2.6.0-preview1, with new Video Source APIs that significantly improve the performance of streaming ReplayKit content. You can try our updated example now.

Best,
Chris

[yahmad]

@ceaglest, some nice improvements there! I will try out the changes asap.

Thanks,
Yasir

[ceaglest]

Hi @yahmad,

In our 2.10.0 update, a ReplayKit extension can connect without subscribing to any Tracks. As long as a Group Room is used, the extension will now require less memory and CPU than before.

Since we haven't heard from you in a while, I'll close out this issue. If you're still having memory usage problems with the latest update please reopen and let us know.

Best,
Chris