Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VP8 is the only supported video codec #88

Open
ceaglest opened this issue Mar 1, 2017 · 23 comments
Open

VP8 is the only supported video codec #88

ceaglest opened this issue Mar 1, 2017 · 23 comments
Assignees
Labels

Comments

@ceaglest
Copy link
Member

@ceaglest ceaglest commented Mar 1, 2017

At the moment our SDKs only support the VP8 video codec. On Android this is hardware accelerated (where possible), but on iOS VP8 support is provided via a software implementation.

On the positive side VP8 support in Chromium WebRTC is mature and well understood. However, performance could be better and ultimately using the hardware codecs (which ship with all iOS devices) is going to provide the best performance for small scale Rooms.

We are working to add support for H.264 on both of our mobile platforms. This ticket will be updated with more details as we get closer to releasing this feature.

@jondwillis

This comment has been minimized.

Copy link

@jondwillis jondwillis commented Jun 3, 2017

Ooof! I am disappointed with vanilla WebRTC performance on iOS, and was looking to try the Twilio SDK instead, but VP8 software decoding is a bit of a blocker for my use case (I have a lot of chrome and other CPU intensive processes going on during video calls.)

VP9 is an optional compilation flag when building WebRTC for iOS, but I could not get it to perform well on anything but an A10 ↔ A10 processor call due to software decoding, borderline on A9 calls.

h.264 main profile is standard and performs reasonably well and similar to VP8, but with reduced CPU strain and power consumption.

h.264 high profile is also available in iOS WebRTC by toggling a field trial flag at runtime, and in my tests performed well all the way down to A6 processor (iPhone 5 and 5C), and perhaps older. It also can save ~20% network bandwidth over main profile h.264.

I realize that cross-platform support for these different encodings may be more difficult, but is it possible that you could allow them as opt-in via SDP?

@jondwillis

This comment has been minimized.

Copy link

@jondwillis jondwillis commented Jun 6, 2017

With iOS 11, native HEVC is available. HEVC Support would need to also be added to WebRTC. Also in iOS 11, Safari is adding WebRTC support, so perhaps there will be patches to WebRTC that enable HEVC. https://twitter.com/fr3ino/status/871801770149900291

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Jun 7, 2017

Hi @jondwillis,

Thanks for your feedback. We have been testing VP9, and H.264 internally and generally agree with your assessment. I don't think we will be using the software VP9 encoder any time soon on iOS as the performance is pretty poor, but H.264 provides some significant performance wins. One thing to note though, is that hardware H.264 encoders and decoders are a limited resource. It's possible for other apps and/or Airplay to use them in parallel with your video app.

The first place we would like to use H.264 is in 1:1 Peer-to-Peer Rooms. We are still considering how to offer this feature to developers (possibly as an opt-in to start, becoming the default in scenarios where it performs well). There are also some implications on the capture pipeline side of things in order to support pixel formats other than NV12 without un-necessary conversions.

As for HEVC, it looks like Safari will start with H.264 support since it is based upon Chromium's WebRTC implementation. Hopefully Apple will consider contributing patches which enable H.265 support for iOS and macOS.

https://webkit.org/blog/7726/announcing-webrtc-and-media-capture/

Now that we've passed 1.0, wider codec support is definitely a priority for our team. I will circle back with our leads and product managers to see where this feature will land in our release schedule.

Regards,
Chris Eagleston

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Jun 25, 2017

Hi,

Thank you to everyone who has provided feedback through support and on Github.

We are targeting Q3 for shipping H.264 support on our iOS Clients, with the goal of having H.264 available for at least some use cases by the time Safari 11 & iOS 11 are released in September. Initially we will be targeting 1:1 Peer-to-Peer Rooms, and a variant of Group Rooms. I will provide more info on this ticket as H.264 rolls out in beta form.

Regards,
Chris Eagleston

@jondwillis

This comment has been minimized.

Copy link

@jondwillis jondwillis commented Jul 9, 2017

@ceaglest Is there a way to use H.264 with Twilio room servers today? Even if it is just peer to peer, this would help me out on my current project greatly.

I am currently using my own room server (not scalable), using Twilio NTS and WebRTC with H.264 as the default.

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Jul 10, 2017

Hi @jondwillis,

This is not possible at the moment with an iOS Twilio Video Client. Our capture pipeline is not strictly compatible with the H.264 encoder so our codec factories don't vend H.264. There is nothing in Peer-to-Peer Rooms that prevents using H.264 but there are also no APIs to promote that codec to the top of the rtp map. Some Android devices support H.264 and offer it today, but the codec is never selected. Same with JavaScript, but you could promote it to the top by modifying twilio-video.js.

Regards,
Chris Eagleston

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Sep 11, 2017

Hello,

A quick update here. We've just released 2.0.0-preview1 which both supports H.264 and allows you to prefer this codec over VP8 at Room connection time.

https://www.twilio.com/docs/api/video/changelogs/ios#200-preview1-september-11-2017

Our sample apps don't yet demonstrate the usage of the new codec selection APIs, and we are still working on updating our guides for the preview release. However, you can get started right now with updated 2.0.0-preview1 compatible QuickStart examples.

https://github.com/twilio/video-quickstart-swift/tree/2.0.0-preview

More updates to come as we flesh out samples and supporting materials. Future preview/beta releases will continue to improve functionality when it comes to error handling, and edge cases regarding H.264 & codec selection.

Best,
Chris Eagleston

@jondwillis

This comment has been minimized.

Copy link

@jondwillis jondwillis commented Sep 13, 2017

@ceaglest Great! This is actually right in time for me to check out before I launch an app. Is there any way to support high profile H.264?

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Sep 13, 2017

Hey @jondwillis,

Our 2.0.0-preview2 release does not support high profile H.264 yet, but we would like to enable it as another TVIVideoCodec option during the preview/beta period. When we do enable high profile, I expect it will only be supported within Peer-to-Peer Rooms to start. In Group Rooms, and without transcoding most platforms are limited to the Constrained Baseline profile and will not be able to decode it.

From your earlier messages, it sounds like P2P support is okay for your use case.

Best,
Chris Eagleston

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Sep 23, 2017

Hi,

For anyone trying out our 2.0.0-preview releases we've updated the Swift QuickStart examples to demonstrate both audio and video codec selection. Please note that Group Rooms currently support the VP8, H.264, and OPUS codecs. If you wish to select H.264 then you need to create your Room via the REST API, otherwise you will get VP8 by default.

We will be publishing an updated guide soon which provides more details on the codec preference APIs. Until then, if you're working with Safari 11 and twilio-video.js see our Developing for Safari 11 guide.

Have a good weekend,
Chris Eagleston

@jondwillis

This comment has been minimized.

Copy link

@jondwillis jondwillis commented Sep 26, 2017

@ceaglest Thanks so much for this! I am currently integrating Twilio Video into my app now that it supports hardware accelerated H.264 on iOS.

You mentioned that high profile H.264 might be in the works, and I know it is easily enabled in WebRTC clients behind a field trial flag. Do you have any idea if and when that might get prioritized?

Also (and this is really a long shot), now that newer devices running iOS 11 and macOS 10.13 support HEVC, and WebRTC supports injectable codecs, do you think it'd be possible to see HEVC support from Twilio p2p rooms in the not-too-distant future? I think with the politics behind WebRTC it might be years before we see mainstream support for it, but I don't see a really good reason (other than some added complexity) not to take advantage of the huge bandwidth wins it provides. https://bugs.chromium.org/p/webrtc/issues/detail?id=8265 Support could be a huge competitive advantage for Twilio since it would bring FaceTime-quality video support for Apple products.

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Sep 26, 2017

Hey @jondwillis,

You mentioned that high profile H.264 might be in the works, and I know it is easily enabled in WebRTC clients behind a field trial flag. Do you have any idea if and when that might get prioritized?

I don't have an estimate to share today, but our team is gathered for quarterly planning this week so we will look at how it fits into our roadmap. I can certainly enable the field trial flag to get this feature working in Peer-to-Peer Rooms without much effort, and provide support for prioritizing the codec over constrained baseline H.264. Where it fits in with the rest of our product, including Group Rooms is something that begs further discussion.

Also (and this is really a long shot), now that newer devices running iOS 11 and macOS 10.13 support HEVC, and WebRTC supports injectable codecs, do you think it'd be possible to see HEVC support from Twilio p2p rooms in the not-too-distant future? I think with the politics behind WebRTC it might be years before we see mainstream support for it, but I don't see a really good reason (other than some added complexity) not to take advantage of the huge bandwidth wins it provides. https://bugs.chromium.org/p/webrtc/issues/detail?id=8265 Support could be a huge competitive advantage for Twilio since it would bring FaceTime-quality video support for Apple products.

I've seen the feature request in Chromium WebRTC, and would be overjoyed if Google were to implement H.265 support. It's certainly got my star, but Twilio doesn't have plans to support H.265 separate from Google's efforts at this time. For what it's worth the VideoToolbox side of things is not much different than H.264, but the packetization and SDP support may be more challenging.

Regards,
Chris Eagleston

@mcorner

This comment has been minimized.

Copy link

@mcorner mcorner commented Oct 18, 2017

@ceaglest Looking at the documentation here: https://www.twilio.com/docs/api/video/rooms-resource#create-room for creating rooms with another codec, it doesn't see like you can say you support more than one? In the client you can list an order of preferredCodecs, but on the on the server side it implies you can have only one.

We will upgrade our clients to 2.0, but not necessarily all at once. It seems like if we switch room creation to just H264 then old clients won't be able to connect?

@piyushtank

This comment has been minimized.

Copy link
Collaborator

@piyushtank piyushtank commented Oct 18, 2017

@mcorner Chris is out on a vacation. You are correct, based on our current feature set on 1.x TwilioVideo SDK, if a Group Room is restricted to use H.264 codecs only, the existing 1.x clients won't be able to negotiate any video codecs in that Group Room.
Thanks for bringing this up, we are discussing this issue. One solution we are discussing internally in the team is backporting H.264 and codec preference feature into 1.x. I will update the ticket when I have more information on this. Thank you!

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Nov 4, 2017

Hey @mcorner,

@ceaglest Looking at the documentation here: https://www.twilio.com/docs/api/video/rooms-resource#create-room for creating rooms with another codec, it doesn't see like you can say you support more than one? In the client you can list an order of preferredCodecs, but on the on the server side it implies you can have only one.

We're working on enabling support for multi-codec Group Rooms. This means that you will be allowed to select both H.264 and VP8 as valid video codecs when you create a Room via the REST API, or via default Room settings (in the console). In multi-codec Rooms client side video codec preferences will still determine which codec you use to publish your video Track.

In Peer-to-Peer Rooms any codec is allowed and its up to the Clients to negotiate a codec which is supported by both parties.

We will upgrade our clients to 2.0, but not necessarily all at once. It seems like if we switch room creation to just H264 then old clients won't be able to connect?

If you choose just H.264 today then older 1.x Clients will be able to connect, but they won't be able to receive any (h.264) video Tracks published in the Group Room. Since they still support the OPUS codec any audio Tracks from remote Participants will be available. In a Peer-to-Peer Room 1.x and 2.x Clients will negotiate a commonly supported video codec (usually VP8).

We're working on a guide which has much more information about codec selection. Hopefully I will have more to share once the guide and hybrid codecs feature goes live.

Best,
Chris Eagleston

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Nov 18, 2017

Hi,

Just a quick update about Group Rooms and our 1.x iOS clients. Earlier I said:

If you choose just H.264 today then older 1.x Clients will be able to connect, but they won't be able to receive any (h.264) video Tracks published in the Group Room. Since they still support the OPUS codec any audio Tracks from remote Participants will be available.

This is the expected behavior. However, there is currently a bug with negotiating a PeerConnection in H.264 Group Rooms where the Client doesn't support H.264 video. You may notice that 1.x Clients do not receive any media and see a logged error:

PeerConnectionSignaling: Client is unable to apply a remote media description 53402

We're currently testing a fix for this issue on the server side. We plan to deploy the change next week, at which point VP8 only Clients will be able to join H.264 Rooms in an audio-only capacity.

Cheers,
Chris Eagleston

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Nov 29, 2017

Hello,

This morning we deployed a new release of our Media Servers which resolved the 53402 errors for 1.x Clients.

Best,
Chris

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Feb 7, 2018

Hi,

We've recently published a guide on Managing Codecs which may provide some helpful advice for those working with VP8, VP9, H.264 and audio codecs.

Amongst other news, we're working on VP8 Simulcast support for our mobile SDKs and we haven't forgotten about H.264 profiles. We should have more to share about this in the coming weeks as we enter the beta phase for our 2.0 mobile SDKs.

Best,
Chris

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Jun 21, 2018

Hello,

As of 2.1.1, we now support VP8 simulcast on iOS. We have also went ahead and updated our example code to allow you to try out VP8 simulcast for yourself.

#273

Best,
Chris

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Jul 3, 2018

Hi folks,

I've got another update for you on video codec support.

As a part of transitioning our mobile SDKs to WebRTC 67 we have been testing scenarios involving multiple H.264 profile levels. During this testing, we discovered that our mobile clients don't always handle codec preferences correctly when multiple profile levels are present in an SDP. We are working on resolving the codec preference issue in our current sprint, and once we do we will be able extend TVIH264Codec to select the Constrained High profile. More info to come on this one.

For anyone who is using H.264 today, you might find our new VideoRenderer example interesting. It demonstrates rendering of remote video using AVSampleBufferDisplayLayer, something which isn't as practical when your source is a VP8 decoder. We are putting the finishing touches on the Pull Request and expect to merge it next week.

Best,
Chris

@coderzone014

This comment has been minimized.

Copy link

@coderzone014 coderzone014 commented Aug 6, 2018

@ceaglest Good day! How can I change the order of video codecs? by default its VP8/VP9, H264, I need to re-order it into H264 VP8/VP9, reason is memory usage of IOS spikes when it receives either VP8/VP9 video codecs. Hope to hear from you.

Thank you,
Jef

@piyushtank

This comment has been minimized.

Copy link
Collaborator

@piyushtank piyushtank commented Aug 6, 2018

@coderzone014 Chris is out on Monday, Aug 6th. He will be back on Tuesday, Aug 7th.
You can change the codec preference by setting the preferredVideoCodecs property on TVIConnectOptions. [Here] is the exmple on how to do this. Also, our QuickStart demonstrates this feature where you can chose the preferred codec from Settings (top right on screen).

@ceaglest

This comment has been minimized.

Copy link
Member Author

@ceaglest ceaglest commented Aug 6, 2018

Hi @coderzone014,

Just a quick note, if you are concerned about the exact ordering of codecs you can go one step further than the example and specify preferences for all video codecs like so:

let connectOptions = TVIConnectOptions.init(token: accessToken) { (builder) in
    // In order of preference - H264, VP8, then VP9
    builder.preferredVideoCodecs = [TVIH264Codec(), TVIVp8Codec(), TVIVp9Codec()]
}

That's it for me, catch you tomorrow.

Best,
Chris

paynerc added a commit that referenced this issue Jun 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.