Codec order preference for Video and Audio tracks

[j0hnlam]

Have you read the FAQ and checked for duplicate open issues?
Yes

Is your feature request related to a problem? Please describe.

For dash manifests with multiple codecs for audio and video, Shaka player currently chooses the lowest bitrate bandwidth codec. I would like to have the ability to override that by sending an array of codecs to the shaka config.

Describe the solution you'd like
I would like the ability to pass an array of codecs that I want Shaka player to consider as my preference when choosing a codec.
videoCodecPreference: ['hevc', 'avc', etc]
audioCodecPreference: ['ac-3', 'mp4a.40.2', etc]

Describe alternatives you've considered
For audio codec, I considered the preferredAudioChannelCount shaka config but there are times where the manifest does not provide AudioChannelConfiguration channel count so that config property does not work.

For video, I have not able to find an out of the box support in the current Shaka player to let me choose what video codec to use.

Additional context
This is to be used on the Chromecast receiver where I will first check the canDisplayType method in the CAF Receiver SDK and then populate the codec preference array that will be passed to the Shaka player to choose.

It allows Chromecast users the ability to have a higher bitrate codec viewing and audio experience.

[joeyparrish]

It allows Chromecast users the ability to have a higher bitrate codec viewing and audio experience.

Why would you want to consume more bandwidth? In your example, you pass hevc ahead of avc, which would correspond to lower bitrates at the same resolution, would it not?

[j0hnlam]

In the manifest that I am provided, that is not the case. HEVC will only have 4K, 1080p, while avc will only have 1080p and lower.

<Representation id="hevc-ffa5v1-24p-primary-28800000" codecs="hvc1.1.6.H150.90" mimeType="video/mp4" width="3840" height="2160" bandwidth="28800000"></Representation>
<Representation id="hevc-ffa5v1-24p-primary-7200000" codecs="hvc1.1.6.H120.90" mimeType="video/mp4" width="1920" height="1080" bandwidth="7200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-7200000" codecs="avc1.640028" mimeType="video/mp4" width="1920" height="1080" bandwidth="7200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-3200000" codecs="avc1.64001f" mimeType="video/mp4" width="1280" height="720" bandwidth="3200000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-1800000" codecs="avc1.64001f" mimeType="video/mp4" width="960" height="540" bandwidth="1800000"></Representation>
<Representation id="h264-ffa5v1-24p-primary-800000" codecs="avc1.64001e" mimeType="video/mp4" width="640" height="360" bandwidth="800000"></Representation>

[joeyparrish]

I see. And what if the user's internet connection is too poor to sustain the 1080p bitrate, but we've chosen hevc based on the codec preference before we had any bandwidth estimates?

[j0hnlam]

It's one of the risk I accept that a user can run into. However on my end, I allow the user to select the quality (ultimate, best, good) preference before establishing a cast session.

So if a user is experiencing buffering constantly, we suggest the user to lower their quality setting on the app (a message popup on receiver) and restart their Chromecast session.

[riksagar]

Hi @joeyparrish , we have a similar situation. We would like the player to prefer Dolby Digital audio over AAC , but there's no way to express a preference in the config. Shaka seems to pick one arbitrarily.

Sometimes the AAC and DDP bitrates are the same in the manifest. Often the number of channels is the same, or not reported in the manifest.

There's no good workaround. Can you provide an API we can call to force the preferred adaptation(s). We could use that once you've parsed and filtered the manifest and compiled the list of supported adaptations/variants.

[joeyparrish]

Today, you can force a particular resolution or bitrate through the tracks API (getVariantTracks(), selectVariantTrack()), but the list of tracks is fixed once we choose a codec.

We can definitely provide a way to specify a preference. Thank you both for explaining your use cases for me. This will definitely help.

This relates to our adoption of MediaCapabilities (#1391), which will open up additional ways for us to choose a codec (based on bandwidth, on smoothness, or on power efficiency). Having one more option to use an explicit preference list will be relatively simple to add at that stage.

This also relates to our adoption of codec switching (#1528), which will allow us to avoid choosing a specific codec if the platform will allow us to adapt between codecs.

[sunnyoui]

@joeyparrish Is it possible to check the supported codecs on the platform using Shaka API before passing a manifest to the player? Secondly, is Shaka using hardware or software decoding on chromecast hardware?

[TheModMaker]

We already should filter out unplayable codecs. We check the platform for support before playing the streams. So you should never see any streams that cannot be played.

We don't do anything special for software vs hardware support. We just give the data to the browser and it plays it however it wants.

[joeyparrish]

Yes, JavaScript apps have no direct visibility into hardware vs software codecs. We get a hint with MediaCapabilities, in which hardware codecs should generally be labelled as "power efficient". But some software codecs could also be labeled as such, for low res or low bitrate. So there's no direct way to tell the difference from a web app.

[naktinis]

I wanted to also add that selecting priorities would be great for our use-case as well. We first generate avc, because it's faster and supported by more browsers (i.e. Safari) and then we generate vp09, because it has better efficiency. I'd like to be able to configure Shaka player to prefer vp09 whenever supported and fall back to avc when either vp09 is not supported, or it's not in the manifest.

[joeyparrish]

If your preference is for the most efficient codecs, this is what we already do today. We base this on the average bitrate in the manifest. So if your vp9 streams are lower bitrate (as they should be for consistent quality at a certain resolution), then you already get this behavior.

If you have consistent bitrates but different resolutions per codec, this is where an explicit codec preference order would help. Or maybe you have some other reason to force us to prefer a certain codec. The mechanism for that would be agnostic to your reasons.

With MediaCapabilities, we would be able to offer preferences based on other things, such as battery life or dropped frames.

[naktinis]

@joeyparrish thanks for the clarification. I think the issue is that the vp9 with our settings is not guaranteed to have a lower bitrate. The idea is roughly that we use avc mainly as a fallback, so it's ok for it to sometimes have lower quality and lower bitrate. Anyway, just wanted to add that an explicit list would also be helpful in our case.

[joeyparrish]

Understood. I'll make sure this is part of our MediaCapabilities planning, as that is when we plan to introduce a set of preferences for choosing codecs.

[mp4allday]

Would ac3 audio stream type be supported by shaka player running on tizen tv?

[joeyparrish]

Codec support is a matter of what the browser supports, not what the player supports. We don't provide any decoders in JavaScript. :-) Please see #2661 for a more detailed explanation.

[mp4allday]

Thanks for the note @joeyparrish. Most tizen tvs if not all support AC3 decoding. I am afraid that when shaka is parsing the DASH manifest it is not able to select the ac-3 adaptation set. Is it a mime type related issue?

[joeyparrish]

Check https://shaka-player-demo.appspot.com/support.html on the device to see what the browser reports support for. This is what we base codec decisions on.

[MilosRasic]

Found this issue while investigating before opening one. Our need for this feature is specific to audio codecs. We have ec-3, dtse and mp4a listed in our manifests. ec-3 and dtse both have 6 channels and same bitrate specified. The relevant tags look like this:

dtse

    <AdaptationSet
      group="1"
      contentType="audio"
      lang="en"
      segmentAlignment="true"
      subsegmentAlignment="true"
      subsegmentStartsWithSAP="1"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="dtse"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:dts:dash:audio_channel_configuration:2012"
        value="6">
...
      <Representation
        id="audio_eng_1=384000"
        bandwidth="384000">

ec-3

    <AdaptationSet
      group="1"
      contentType="audio"
      lang="en"
      segmentAlignment="true"
      subsegmentAlignment="true"
      subsegmentStartsWithSAP="1"
      audioSamplingRate="48000"
      mimeType="audio/mp4"
      codecs="ec-3"
      startWithSAP="1">
      <AudioChannelConfiguration
        schemeIdUri="urn:dolby:dash:audio_channel_configuration:2011"
        value="F801">
...
      <Representation
        id="audio_eng_2=384000"
        bandwidth="384000">

We prefer dtse for various reasons and in cases like this when codecs have matching channel count and required bandwidth, Shaka seems to choose one by inertia, which always ends up being ec-3 in our case.

We'd like to have a way to specify dtse as preferred codec to serve as a tie breaker in cases like this and can spare the bandwidth to contribute a feature, as long as we can come up with an acceptable API.

I'd suggest preferredAudioCodecs as an array of codec identifiers, with element at index 0 preferred if it's found in the supported codecs, then index 1, etc. It would have lower priority than bitrate and preferredAudioChannelCount and only be used when preferring by closeness to the specified channel count and lower bitrate yields more than one result.

Meanwhile @riksagar, any details you could share about your workaround? 🙂

[riksagar]

Hi @MilosRasic . Well it's not a pretty fix!

We basically added code to modify the manifest prior to Shaka getting it. We remove the AAC Adaptation, then Shaka is forced to choose DDP (this is assuming the platform supports DDP. We also need to perform that check too).

This is all done in the networking engine. We register a response filter callback (see player.getNeworkingEngine().registerResponseFilter()) and for the MANIFEST request modify the DOM. The manifest DOM is in the data property of the response.

[MilosRasic]

Thanks @riksagar . Agreed, not pretty at all, but good to know it works.

[dsparacio]

Hi @joeyparrish Glad to see this in milestone 3.2. Maybe I can get one of my engineers to contribute to this and we can move it up to 3.1 if done in time? This is a new priority for us ;)

[joeyparrish]

I believe this will be done as part of @michellezhuogg's work on MediaCapabilities. See https://github.com/google/shaka-player/blob/master/docs/design/codec_preferences.md There's already quite a lot of work in progress in this area.

I don't think v3.2 is likely to be far behind v3.1, but I'll leave it to Michelle to decide whether and how you could contribute if you're interested.

[dsparacio]

@joeyparrish thanks for this update, we will follow that and reach out to @michellezhuogg

[dsparacio]

@michellezhuogg. This is high priority for us at ViacomCBS, wondering if there is anything we can help with? How far along are you with this enhancement?

[michellezhuogg]

Hello @dsparacio ,
Here's the currently status:
Phase 1 -- Using MediaCapabilities.decodingInfo() instead of MediaSource.isTypeSupported() is mostly done and is in the master branch. We're launching v3.1 soon, and we're planning to implement Phase 2 -- providing configurations for codec preferences after v3.1.

Meanwhile, you're welcome to send us pull requests for that! Let me know if you have any questions. Thank you!

[dsparacio]

Hi @michellezhuogg Thanks, we will take a look and see if it is feasible for us to fully implement the filtering part.

We just hit an issue on Playstation 5, where they did not implement MediaCapabilities.decodingInfo in their stack. So we have to check for this. Just passing that along as it is related to this discussion.

• PS5 does support hevc and ac3 however so this first phase approach may have some holes in it.

Otherwise,
we have had great playback success with HEVC and AC3 manifest on LG, Samsung, and Vizio so far via Shaka 3.x. So this is exciting,

I think your idea of supplying a list of codecs in the order shaka should try them is the most flexible approach. I think we just need to learn where the filtering is happening in the code and look at the LOE. Just wanted to touch base, no action or commitment from us just yet. Get back to you soon if we can work on this in our next sprint. Thanks for the info.

[dsparacio]

@michellezhuogg we are looking into phase two, filtering by preferred codec config, to understand how quickly and the LOE of effort to see if we can do this work or help. Will follow up. Looking at 3.1. release to start this.

[michellezhuogg]

Update: the code change is in review now and would be ready for testing soon :)

[dsparacio]

@michellezhuogg that is awesome and we can dedicate a lot of testing on this. Please let me know when we can start.

[avelad]

@michellezhuogg This should be closed as it is included in the master branch, or is something missing?

[joeyparrish]

This work is complete, at least until the first bug reports come in. :-)

[dpaou]

Video / Audio codec preference as shaka configuration setting is documented? Cannot finding anything related to this one. Is it possible to set the audio codec preference for example at latest shaka release and if yes how we can do this?