v3.26.1

Release v3.26.1 (2021-09-14)

The v3.26.1 release has now been published.

It is a relatively small release which provides minor improvements and works around rare browser issues:

• We encountered very rare situations where the player would indefinitely freeze due to some contents and browser issues.
  We now work-around those by automatically "unfreezing" the player when it froze for a given time.
• Likewise a subtle browser issue could in rare scenario lead to media data being requested multiple times in a loop.
  The player now detects this situation and prevent that loop from happening
• We greatly improved the DASH_WASM experimental feature (which accelerates and improves the memory situation when parsing large DASH MPD by relying on a WebAssembly parser).
  Mostly its size and the time to compile it have been drastically reduced.
• More types are exported through the rx-player/types path.
  Those were types needed from already exported APIs, such as types about track preferences.
• Infinite rebuffering issues that could happen when some Periods in a multi-Period contents had no video content should now be fixed
• We continued our efforts to improve on initial loading time, by requesting what is called the initialization segment parallely to the first media segments when it makes sense.

As those are mostly very technical improvements, we will try to explain and dive into the details of what this release brings in the next chapters.
If this is too much details, you can jump as always to the changelog at the end of this release note!

Better handling of "frozen" playback

This release works around a very rare issue we infrequently encountered where the content appears to be indefinitely rebuffering.
This case is internally called "freezing".

How content playback basically works with the RxPlayer

To introduce what we call "frozen" playback, we need to introduce the basics of how content playback with the RxPlayer works.

First, the RxPlayer does not decode audio and video data itself, it relies on the browser (and underlying codecs) to do so.
What happens is that the player fetches the right audio and video "segments" (small decodable parts of the content) and indicates to the browser the position that should be played.
The browser and the codecs it integrates then ensure that the audio and video content corresponding to that position is decoded.

To ensure smooth playback, the player has to make sure that audio and video segments are "pushed" to the browser before the decoding operation reached their corresponding starting positions (or said another way: before we need one of them).
If the player couldn't load the segment before that deadline, the browser won't have anything to decode and will sort-of "freeze" until that data is pushed.

That behavior is also called "rebuffering" or more colloquially just "buffering".

Example of rebuffering. We can see, below the video, a Buffer content chart. It indicates that the current position (the red line in both buffers) is close to the end of the currently buffered audio and video data, we're thus awaiting new data before being able to play

"Freezing"

Yet in rare circumstances, that freezing situation can happen even when the right segments have been "pushed".
There can be legitimate reasons, like when a needed decryption key has not been obtained yet. Though there are times when there seems to be no apparent cause, at least from the point of view of the player.

In the RxPlayer team, we call that problematic case "freezing" (whereas regular rebuffering cases are just internally called "rebuffering").
Thankfully, this happens in very rare circumstances. We usually encounters unexplained stucked playback only in the following cases:

• poorly-packaged contents
• playing with a very high playback rate (i.e. at a high speed)
• when the player is doing risky behavior in some particular platforms - for example pushing a video segment containing data for the already-buffered current position on some old samsung TVs could lead to that situation.
  We however worked hard on this last category to avoid them in most cases.

Here we can see, that the content seems to be rebuffering despite having both audio and video data in front of the current position (the red lines in the Buffer content chart). Here this actually happened after I set a playback rate of 10 (meaning I'm playing the content at 10 times its regular speed) and waiting.

Even if very rare, it can happen and having a logic in last-resort would not be superfluous.

Going out of that frozen state back into regular playback is usually pretty easy.
In most scenarios, a simple little "seek" of 1 millisecond (which means going 1 millisecond in the past or in the future) is sufficient to "unfreeze" playback.
This is probably the case because seeking re-triggers some lower-level code which was linked to the source of freezing, e.g. low-level audio and video buffers could be emptied and filled again and the lower-level playback logic could be restarted from the new "seeked" position.

The new logic

So in definitive, the resolution of that issue is simple: if the player detects that "freezing" is happening, we just have to perform a little seek to unfreeze.

However, detecting when the player is abnormally frozen is not really straightforward.

The naive way would be just to look at two different point in time when playing with enough data, and see if the current position has evolved. If it did not, it must mean that we're stuck and - because we have data - playback must be freezing.

Yet there are multiple issues with doing just that:

1. Poor performance could lead to false positives.
   For example, we noticed that after a seek, some low-end devices can take up to multiple seconds to restart playback, even when enough data is available to play.
   By resolving freezes the naive way, the player thus might think it is frozen because a seek operation is taking a long time to finish. Here it might perform another seek, which will lengthen even more playback (we will also risk to be in an infinite seeking loop !).
2. There are sometimes good reasons for playback not restarting yet. The main one being that the decryption key for the incoming data is not yet obtained.

When considering all possibilities, this "freezing" state thus becomes very complicated to detect.
We ended-up relying on an imperfect yet sufficient solution:

• The player only performs the seek if playback is abnormally stuck after a very high delay (between 6 and 7 seconds).
  Being frozen for around 7 seconds sounds like a lot, but this logic should only run as a last resort anyway.
• A lot more playback characteristics than just the current position are looked at to deduce if we should be able to play.
• Already-existing and less risky "unfreezing" strategies will run before it.

On the same case of freeze while playing at a very high playback rate, we can see that this logic is finally able to un-stuck us.
Here is a case with that logic in action (with added logs):
https://user-images.githubusercontent.com/8694124/124958710-bf687e80-e01a-11eb-94a8-3c2ebaad7301.mp4

Removal of the cached segment detector

Web browsers maintain a somewhat complicated HTTP cache to avoid requesting multiple times already-fetched resources.

Thanks to it, a request made normally from JavaScript (through either an XMLHttpRequest object or a fetch call) will sometimes avoid the round-trip to the server. The main advantage of doing that is that it reduces latency.
All this happens without the knowing of the RxPlayer, for which the request still appear as if it was performed through the network.

Yet detecting when a response comes from the cache is sometimes needed by the player.
For example, the internal logic calculating the network bandwidth (to select the optimal video and audio qualities) relies heavily on the measure of the time it takes to download data from the network. When such data comes from the cache instead, it appears that the user has an exceptionally high bandwidth.

So until this release, the RxPlayer had what we called a Cached Segment Detector, whose job was to deduce heuristically whether data came from the browser cache or the network. It basically compared times to download such data and ignored all metrics which indicated an abnormally high bandwidth.

  -----------------
 | network request | 1. A network request metric is generated
 |     metric      |    after a request
  -----------------
         |                         2. "does it come from
         |                            the browser's cache?"   +----------------+
         +-> +------------------+ --------------------------> | Cached Segment |
             | network listener |               3. "yes"/"no" |    detector    |
             +------------------+ <-------------------------- +----------------+
                     | 4. If it doesn't seem to come from the cache
                     |    (and other rules)
                     |          +-------------------+
                     +--------> | Network bandwidth |
                                |     calculator    |
                                +-------------------+
                                        | 5. gives bandwidth
                                        |    estimate
                                        V
                                 +---------------------+
                                 | audio/video quality |
                                 |       selector      |
                                 +---------------------+

A very rough schema of how the cached segment detector interacted with the rest.
This is not strictly the exact architecture but the spirit of it is here!

Sadly, we found out recently that in some rare conditions (a fast network and a well-compressed and short content basically), the detector had a high rate of false positives. Said another way, the detector wrongly believed that most data came from the cache, when it did in reality come from the network.
In turn this led to a wrong bandwidth calculation and thus to lower video and audio quality than expected.

To resolve that issue, we measured the pros and cons and the alternatives (like never relying on a cache) and finally ended up just removing the detector from the code.
Rest assured, this should not have a huge influence as data is rarely obtained from the cache anyway (as it is rarely re-loaded) and as the RxPlayer handles network bandwidth fluctuations (which might happen when only some data comes from the cache) really well.

Detection of browser-side media data cropping to avoid request loops

Like frozen playback, we worked around another very rare issue whose source seems to be in the browser.
That issue could lead to media data being requested in a loop.

Media garbage collection

As you might have guessed, video and audio data can take a lot of place in memory.
This can become problematic when running a player, because it usually wants to load in advance a large amount of data to ensure a smooth playback, meaning a large amount of memory will be taken by that pre-buffered data. Worse, the RxPlayer also keeps by default already-played data in the buffers it maintains to allow the user to quickly seek back in the content, for example to re-watch a scene or hear a recently-spoken sentence again.

Yet data cannot be kept indefinitely as memory is not infinite, so "old" media data needs to be removed at some point in time.
By default the RxPlayer let the browser decides when to collect media data and which media data should be removed - because the browser generally has the better view on memory usage (relatively to the JavaScript environment the RxPlayer is a part of).

Here (in this poorly-cropped looping gif!), we can see in the Buffer content chart that as we play, the beginning of the video buffer seems to be progressively removed.
This is a behavior performed by the browser (Chrome in this case).

This mechanism of media data removal is called "garbage collection".
It might sound familiar to JS developers, yet it is not exactly the same thing than the "garbage collection" concept generally encountered in programming languages.
Here, media data removal can be performed even if the user still could want to play the corresponding data. In that last scenario, the RxPlayer would need to load that data again.

Problems with garbage collection

The garbage collection logic performed by the browser is opaque. It is done following its own set of rules, and the browser won't directly tell the player which media data has been removed as well as when it removed it.

This is problematic because the RxPlayer needs that information to know which media data should be loaded next. For example if garbage collected data is needed again, the RxPlayer should notice that it has been garbage collected and then re-download that data.

Here we're seeking in an old garbage-collected part of the video buffer.
The RxPlayer here knows it has to re-load the corresponding data. We can even see video data at the end of the buffer now being garbage-collected by Chrome (as there is now nothing "old" to collect).

In the end, instead of explicit communication with the browser on garbage collection, the RxPlayer tries to deduce when it happened.
It can do this thanks to a browser API allowing to known at any time which position contains data in the browser's media buffers and which doesn't.
The RxPlayer then maintains its own inventory (called the Segment Inventory for various reasons) which compares the data the player expects to be buffered to the data actually buffered by the browser.
When the browser's buffer contains less data than the player's inventory, the RxPlayer can deduce that garbage collection happened and re-download the corresponding data.

Risks of request loops

This strategy works generally very well.
Yet, there was still an issue that could arise when the RxPlayer's inventory and the browser's buffer had different opinions on the data buffered.

We've recently seen this case in Safari. Here audio data that the RxPlayer's inventory initially guessed would start at 0 second actually started at 0.45 seconds once in Safari's audio buffer. That's a 450 millisecond difference, a big difference.
Because of that difference, the RxPlayer incorrectly guessed that the data was garbage collected and re-loaded it.
After it pushed another time that same data to the browser, the player noticed the same difference and thus scheduled the request another time.

This led to a loop of segment requests.

Comparison of the behavior in Chrome and Safari taken from the aforementioned issue.
By looking at the network information, we notice that on Safari the same media data is being requested in a loop.

New strategy to avoid request loops

As we found our current logic to still be coherent and efficient, we didn't change it and instead worked on detecting abnormal browser behavior (like in the issue with Safari).

This is done by storing a short-lived history of the browser's buffer evolution alongside the RxPlayer's inventory.
When media data appears to be garbage collected just after being given to the browser, the player first looks if that same media data has already been communicated recently and if it has, if the same garbage-collection pattern happened before, also just after pushing the segment.
If it did, the corresponding data is not re-loaded another time.

This means that request for that kind of data is still performed two times: the first time we will consider that the data has really been garbage collected, the second time, we will consider that this is a browser issue and just ignore it (the corresponding hole in the buffer will be automatically skipped).

This situation of still doing two requests can appear to be non-optimal, but we prefer it because request loops are very rare, much less frequent than real garbage collection.

WebAssembly MPD parser improvements

In the previous v3.26.0 release, we introduced an experimental DASH_WASM feature, allowing to speed-up the parsing of large MPD through the usage of WebAssembly.

Because we've seen very good results while using it in production, we continued working on it. This release brings multiple improvements:

1. Its size has been greatly reduced. It went from 424 KB to 152 KB, a ~64% reduction!
2. On low-end devices, its compilation time (a necessary step in the browser) has similarly seen drastic improvments
3. We fixed a small issue that could arise when <EventStream> elements were parsed in the MPD: parsing of such elements could previously fail if they contained namespaced element names or attributes. This led to the corresponding events not being sent. This has now been fixed.

More exported types

The RxPlayer exports TypeScript types through the rx-player/types directory so that a TypeScript codebase can avoid to define its own types when it needs when processing what the different RxPlayer's API takes in argument or return.

We've received some feedbacks that this was a good thing, yet we missed a fair share of complicated types in it.

This release has thus been the occasion to exports the supplementary types from rx-player/types:
- IPersistentSessionInfo and IPersistentSessionStorage: needed for API related to persistent licenses
- IPlayerError: combines the different RxPlayer's errors
- IRepresentationFilter: The representationFilter callback
- IRepresentationInfos: The second argument of the representationFilter callback
- ISegmentLoader: The segmentLoader callback
- IManifestLoader: The manifestLoader callback
- IManifest, IPeriod, IAdaptation, IRepresentation, ISegment: Respectively the Manifest, Period, Adaptation, Representation and Segment structures returned and emitted by several player methods and events

All those types have been added to the corresponding documentation page.

Technically, this improvement could be considered supplementary features and thus a new minor version when talking about semantic versioning.
However, we preferred doing only a new patch version as such change was very minor and could just be considered like missing information we should have had before.

Loading of the initialization segment in parallel of media segments

As written earlier, audio and video media data is actually segmented into multiple decodable parts called "segments".
Those are the units of media data downloaded by the player.

On the left, the video playing. On the right, the list of segments being loaded (here called "fragments").

Those segments actually come in two flavor:

• the "media segments" containing the media data that will be played.
• the "initialization segment", which is loaded before any media segments and provide initialization information to the media decoding logic

To be able to play, the RxPlayer first needs to give both to the browser: first the initialization segment and then the media segment(s).
Before this release, requests were done sequentially: the initialization segment was first loaded and only then, the media segments was in turn loaded.

In this release, we try when it makes sense to perform both requests in parallel.
Note that the segments are still sequentially pushed (else it doesn't work), only requests are parallelized.

This theoretically should lead to better content-loading performances (less delay between a loadVideo call and the playback actually starting), because requests have various overheads which do not combine when performed at the same time (e.g. networks delays, browser-originated scheduling...).

Improved demo page

The demo page is a very important aspect for the RxPlayer project. It allows us to easily test new player features without too much effort and it also permits current and future users to test custom contents.

Moreover, the RxPlayer is highly configurable and tries to address a wide variety of supported environment (smart TVs, STB, Gaming console, Chromecast...) which all have their specificities.
The strategy taken to handle all this is through a highly configurable API, letting the application the possibility to tweak most of the player's behavior to adapt to various platforms and use cases.

Unfortunately until now, there was no real way in the demo page to play with the settings that the RxPlayer exposes. That's the reason why we decided to add a visual "player options dashboard" inside the demo. It will allow the user to play with the API.

Because an image is worth a thousand words, here is the preview of what it looks like:

By default when the user will open the options thanks to the "Show Options" button, all the fields will be pre-filled with their default values.

Then, if the user decides to edit a field, for example, the wantedBufferAhead parameter by a value different from the default value, a small icon will become red, to indicate that the field is no longer to the default value and thus has been "modified".
If you decide later that you want to go back to the default value, you can simply click on that red button to reset to the default value.

Moreover we added a switch button for certain numerical fields that allows to signal that you don't want to limit the corresponding value.

Pending work on the v4.0.0

We profit from this release note to announce that we've also been working recently on a future v4.0.0 release.

It is still far from being finished, but aims to both facilitate the maintenance of the RxPlayer (which didn't have any major version for what is soon 4 years) and to improve the API situation when playing DASH multi-Period contents.

For example, we completely reworked how quality selection API work so that it will be possible to choose a Representation based on its characteristics directly, without needing to identify it by its bitrate.
It will also be possible to select at any time the wanted track and representation for any Period in the content.

Rest assured, we're conscious that switching between major versions can be a pain for application developers, so we'll try to be careful to not change too much the current API and behavior of the player.

Changelog

Bug fixes

• ttml: Do not throw if a TTML subtitles file doesn't contain any <body> tag, just ignore it [#993]
• Auto-detect when playback is unexplicably frozen and try to unlock it through a small seek [#982]
• Properly send available{Audio,Video}BitratesChange event for multi-Period contents [#983]
• DASH/MetaPlaylist/Local: fix rare infinite rebuffering issue which could happen when changing or disabling the track of a future Period [#1000]
• compat: Prevent rare segment-loading loops by automatically detecting when segments are garbage collected by the browser immediately after being pushed [#987, #990]
• compat/DRM: In some Safari versions, communicating a license as a JS ArrayBuffer could throw, this is now fixed [#974]
• DASH_WASM: Don't stop with a fatal error if an expected ISO8601 duration value is empty in the MPD
• DASH_WASM: Parse <Event> elements which contain an XML namespace defined outside that element [#981]
• DASH_WASM: Drastically reduce wasm compilation time and file size [#980]

Other improvements

• Request initialization segment and the first media segments at the same time when possible, potentially reducing loading times [#973]
• Remove cached segment request detection in the adaptive logic, as it is sensible to false positives, leading to a poor bitrate in some short contents [#977]
• Export more needed types through the rx-player/types path [#972, #976]
• demo: Expose some player options in the demo page [#999]
• dev: Rewrite build logic from bash to node.js to improve its maintainability
• dev: Replace internal info script by more helpful and interactive list script [#991]
• dev/code: Forbid the usage of TypeScript's type any in most of the RxPlayer's code - performing runtime type-checking in some cases (in DEV mode only) [#994]
• dev/code: Remove RxJS from the transports code [#962]