Closed Captions (WebVTT) aren't displayed in IE11 in some Windows 10 OS when stream is encrypted (AES-128), cueChange event isn't fired in IE11

[ghost]

When HLS stream is encrypted in some Windows10 instances captions aren't displayed in IE11. I wonder if there is a minimum requirement for OS version to support this feature? Is there any other configuration do we need for IE11/Windows10 to enable captions in the player when the stream is encrypted?

For example in player V0.14.12 in Windows Pro, OS build 19041 (latest Win10 OS) where TPM and PCR7 configuration are available we can see the captions when the stream is encrypted (AES-128). In those Win10 instances where captions aren't displayed we see 'cueChange' event isn't fired in IE11. I wonder if it can depend on "unknown/limited" IE11 support of 'cueChange' event as it is mentioned here: https://developer.mozilla.org/en-US/docs/Web/API/TextTrack/cuechange_event ? Or it can be because of player failure in the decryption of encrypted captions? I enabled the debug log and I don't see any error in the console log. Do we need TPM and PCR7 configuration in Windows 10 to enable captions in this scenario? Please note there is no issue with captions with encrypted HLS stream in Edge, Firefox and Chrome.

Unfortunately, I can't share my encrypted stream, but if it is necessary I can demonstrate the issue via Zoom.

[robwalch]

Hi @porasl,

WebVTT or 608 captions?

I enabled the debug log and I don't see any error in the console log

Try step debugging through subtitle-stream-controller onFragLoaded if you have encrypted WebVTT files. In there you can see if this.decrypter.decrypt and running and what it returns.

After that timeline-controller is where cues get created for either 608 or WebVTT if I am not mistaken. onFragDecrypted would get the decrypted WebVTT or onFragParsingUserdata is where 608 captions are extracted.

[ghost]

Hi @robwalch

It is WebVTT. Encrypted caption works in all other browsers and in all win 10 OS instances except IE11. It works in IE11 only in an instance where TPM is available, and even not in all Win10 OS where TPM is available. Does the player need any hardware module or configuration in Win10 to support caption decryption in IE11? I ask this since the player manages the decryption of captions with no hardware dependency in other browsers and I see logic about it in the HLS JS code.

When I debug my JS code (in those instances where IE11 doesn't work) I get the first cue and track 0 is used, but thereafter no "cueChange" event gets fired by IE11. It sounds like a bug in IE11 in some instances about the support of 'cueChange" event or a configuration value is required to let this IE11 event be fired (in Win10 or IE11) Or there are no captions (because of the player failure in decryption, no trace about the error in console log). I am running Chrome in parallel and I see the stream has captions and it gets displayed on the player in Chrome. And always encoded captions work in IE11 on my own laptop, same Win10 build, same IE11 version, same stream, and at the same time. ( Win10 OS build V19041)

Unfortunately, I am behind the firewall and I can't share my HLS stream, but anytime I can share my screen if it is needed.

I start debugging the player code as you mentioned,

Thank You,
Hamid Porasl

[phloxic]

@porasl - with which content-type are you serving the captions?

[ghost]

@blacktrash
the input stream to my encoder is a non-encrypted akamai live HLS stream from Bloomberg https://liveproduseast.akamaized.net/us/Channel-USTV-AWS-virginia-1/Source-USTV-1000-1_live.m3u8 . And extracting the captions isn't an issue when we don't encode the stream (AES-128) in IE11. When we encrypt the stream then in IE11 (in many instances) in Win10 except in my laptop IE11 we don't see decrypted captions. As I see the main reason is cueChange event doesn't get fired. We have a wrapper JS that retrieves the captions and displays it. I don't see any error in that.
Here in IE11 we see encoded caption streams are received by the player:

In Chrome, we see encrypted captions are received



But in IE11 we don't see captions. Please note we retrieve the captions in our JS code as it was mentioned above, when the browser fires cueChange event and we display captions on a gray overlay on the player as you see in the screenshot. As mentioned only in one Win10 instance it works, same build V19041, same IE version, same stream, and at the same time. here is a side by side screenshot of the payer in 2 different Win10 instances. One on my laptop that works and another Win10 instance ( it is a VM).

@robwalch has recommended to debug the Hls.js code to make sure captions are decrypted. I am planning to do so.
But it will be a huge help to me if you guys tell me this different behavior in 2 different Windows10 instances can be related to any hardware, like missing TPM module or not. Since HLS stream can be decrypted and everything works in other players I believe turning ON TPM isn't necessary for Closed Caption support in IE11 when it is encrypted. Am I right?

In the Chrome Network tab we can see cations are encoded:

[robwalch]

I don't know anything about TPM or how it would impact decryption.

Another sanity check you can do is look at $('video').textTracks. Is there a subtitles track? How many cues does it have? What are their start and end times?

If captions are being decrypted then they would be added to a TextTrack - you mentioned you see one - so are there more but they aren't displayed?

[phloxic]

@porasl - it's a shot in the dark, but can you try to serve the .vtt as text/vtt instead of application/octet-stream? IE11 often stumbles over incorrect mime-type for VTT.

[ghost]

@robwalch @blacktrash
I cleaned the browser cache and I debugged the player and I see when the stream is encrypted in function onFragDecrypted(data) line 19090 the frag type never is equal to 'subtitle'. It shows encrypted VTT content never gets decrypted.

But in another Windows 10 instance which encrypted captions works I see fragment types equal to 'subtitle" are received in the function onFragDecrypted(data). Is there anything else I can debug in hls.js to see why subtitle fragments don't get decrypted?

In the next step I will follow what @blacktrash recommended above about the MIME Type for VTT.

[robwalch]

@porasl So what happens in onFragLoaded > this.decrypter.decrypt? The point of debugging is to determine what the issue is in IE11; at what point does encrypted vtt fragment handling fail?

  onFragLoaded (data) {
//...
    if (this.state === State.FRAG_LOADING &&
        fragCurrent &&
        data.frag.type === 'subtitle' &&
        fragCurrent.sn === data.frag.sn) {
      // check to see if the payload needs to be decrypted
      if (data.payload.byteLength > 0 && (decryptData && decryptData.key && decryptData.method === 'AES-128')) {
        // decrypt the subtitles
        this.decrypter.decrypt(data.payload, decryptData.key.buffer, decryptData.iv.buffer, function (decryptedData) {
          hls.trigger(Event.FRAG_DECRYPTED, { frag: fragLoaded, payload: decryptedData, stats: { tstart: startTime, tdecrypt: endTime } });

  decrypt (data, key, iv, callback) {
    if (this.disableWebCrypto && this.config.enableSoftwareAES) {
      if (this.logEnabled) {
        logger.log('JS AES decrypt');
        this.logEnabled = false;
      }
      let decryptor = this.decryptor;
      if (!decryptor) {
        this.decryptor = decryptor = new AESDecryptor();
      }

      decryptor.expandKey(key);
      callback(decryptor.decrypt(data, 0, iv, this.removePKCS7Padding));
    } else {
      if (this.logEnabled) {
        logger.log('WebCrypto AES decrypt');
        this.logEnabled = false;
      }
      const subtle = this.subtle;
      if (this.key !== key) {
        this.key = key;
        this.fastAesKey = new FastAESKey(subtle, key);
      }

      this.fastAesKey.expandKey()
        .then((aesKey) => {
          // decrypt using web crypto
          let crypto = new AESCrypto(subtle, iv);
          crypto.decrypt(data, aesKey)
            .catch((err) => {
              this.onWebCryptoError(err, data, key, iv, callback);
            })
            .then((result) => {
              callback(result);
            });
        })
        .catch((err) => {
          this.onWebCryptoError(err, data, key, iv, callback);
        });
    }
  }

[ghost]

On my JS code, I removed the listener of FragDecrypted. And I see one hit on

but the start and end time are equal. In the startup, I saw the caption once on the player

Once I left the player on developer mode on debugging state for a long time. then I saw captions started playing. When I left developer mode, it stopped playing. It sounds a question of timing, buffering, or something like that. I keep on debugging ...

[ghost]

@robwalch @blacktrash
Only in few cases (maybe 3 to 5% of cases) as shown above, just once after loading the player VTT file is requested and loaded/parsed. But thereafter caption manifest list is downloaded always, but there is no GET request to get the VTT files:

I have 4 windows 10 instances and 3 have this issue with IE11. When the caption stream isn't decrypted captions are working on all of my 4 windows 10 instances. That is why there is no issue related to downloading .vtt files. All instances are windows pro and they have the latest Windows 10 builds. It seems when captions are encrypted something prevents IE11 to send GET request to get the caption VTT file. What is confusing are 1) in 5% of cases captions get downloaded and they get parsed just once (after loading of the player). 2) In one of my Windows10 instances (on my laptop), it works always. I checked IE11 settings I can't find differences and I even did reset IE11 settings, but the same result.

[ghost]

@robwalch I am behind the firewall and I can't share my encrypted stream with the public. But if you want I can provide private access or I can share my desktop to demonstrate this issue.

@blacktrash I changed the server to serve the vtt's Mime type as text/vtt, but this issue isn't resolved.

[ghost]

I keep debugging the player to see why there is no GET request from the player to load the VTT file, since the list of VTT files are already loaded by the player (this happens in IE11 only when the stream is encrypted:

[ghost]

Here you can see the issue

[ghost]

I tested this issue with HLS JS V0.14.16 and found the issue is resolved. That is why I closed this ticket.