Live Streaming and ExoPlayer Shenanigans #1147

karyogamy · 2018-02-26T06:00:39Z

Added cacheless data source to allow HLS livestream playback.
Added new async loader for media sources to correct HLS livestream loading and enable better control over expired media sources.
Added custom cache key for extractor sources to allow more persistent reuse.
Added seamless shuffling.
Added toggleable fast inexact seeking.
Added toggleable auto-queuing when last stream on non-repeating play queue starts.
Added custom track selector to allow irregular caption text language names, until language normalization NPE in ExoPlayer is fixed.
Added buffering strategy to enable faster playback start.
Fixed inconsistent audio focus when audio becomes noisy through unplugging headsets. Fixes NewPipe start playing music again after beeing paused if a SMS is received #1139.
Fixed search fragment long press dialog deletes item prior confirming.
Fixed potential TransactionTooLargeException when passing player intents.
Fixed main video player state saving on lifecycle events.
Updated ExoPlayer to 2.7.0.
Updated video player UI to no longer show interface after buffering and double-tap seeking. Fixes Don't show overlay after resuming from buffering #1151.
Updated main video player UI to use hidden translucent system UI (nav and status bars), similar to Youtube app.
Updated Extractor to allow HLS extraction.
Removed livestream exceptions.
Removed debouncing time for stream selection on play queue.

Addendum:

HLS playlists really doesn't like loading and playing on demand, which means we can no longer use the existing DeferredMediaSource. Therefore, the old loader is being replaced by a new async loading inside the MediaSourceManager.
It works by first generating a list of placeholder media sources in order to create a playlist media source. When these placeholder are played, they stall the player until they are replaced with a LoadedMediaSource. When loading is requested, outside the lifecycle of MediaSource, the MediaSourceManager loads and builds the MediaSource asynchronously and replaces the PlaceholderMediaSource at the corresponding index on the playlist media source. If the loading fails for a particular media source, then a FailedMediaSource will be used for replacement.
By doing this, HLS livestream is now able to load the playlists properly and we can now have better control on the playlist to evict or update MediaSources with expired urls.
Regarding the CustomTrackSelector, it is mostly a copypasta from DefaultTrackSelector. A few lines are changed in order to allow us to use irregular caption language names. This will be removed once ExoPlayer properly nullchecks their language code normalization.
Currently, I have disabled track selection on livestreams since the manifest allows ExoPlayer to adaptively choose the quality according to the network status. Of course, this adaptiveness can be overriden if needed, but this requires us to create a separate quality menu just for livestreams, which will make the video player more complex. Let me know what you think.

Todos:

Add UI for livestreams to seek directly to live edge.
Change extractor version to latest once merged.

-PoC for new seamless stream loading mechanism.

-Fixed media source manager excessive loading. -Remove unneeded fields in loaded media source.

…headset unplugged). -Fixed live media sources failing when using cached data source by introducing cacheless data sources. -Added custom track selector to circumvent ExoPlayer's language normalization NPE. -Updated Extractor to correctly load live streams. -Removed deprecated deferred media source and media source manager. -Removed Livestream exceptions.

-Extracted expiration time in media source manager. -Re-enabled long click on live stream info items. -Fixed dash source building to use mpd instead of extractor.

…tion policy. -Added sync buttons on live streams to allow seeking to live edge. -Added custom cache key for extractor sources to allow more persistent reuse. -Refactored player data source factories into own class and separating live and non-live data sources.

-Fixed play queue playlist desynchronization caused by media source manager window loading expansion on sublist prior to current item. -Fixed failed media source not treated as ready for playback.

…SourceManager. -Modified LoadController to allow fast playback start and increased buffer zigzag window. -Removed unnecessary loading on timeline changes. -Changed select message in MediaSourceManager to cause immediate load. -Reduced default expiration time in MediaSourceManager. -Fixed main video player not showing end time on audio-only streams. -Fixed live stream has player view disabled after transitioning from audio stream. -Fixed inconsistent progress bar height between live and non-live video on main player.

…ge for intent transactions. -Fixed potential transaction too large exceptions for player intents.

…ger to allow faster access time. -Added some null checks annotation.

karyogamy · 2018-03-01T08:04:53Z

A couple updates:

The player load control is now updated to almost immediately start playback once the buffer is ready. There should be a noticeable performance gain on lower resolutions or a stable connection. After the initial playback start, the buffer should retain the current behaviour of zigzagging between the minimum and maximum buffer time.
I've also experimented with forcing a constant window of buffer, but this may cause the network radio to constantly turn on or off or stay on for the entire duration of the playback, which may lead to more battery consumption. If anyone has a better buffering strategy, please take a look here first and let me know.
The debouncing duration for selecting video or clicking on previous or next button has now been remove. That's 400ms of performance gain, and the user can decide how much they want to load. This reminds me of a story where the project manager tells the programmers to leave some Thread.sleep here and there in the code, so that whenever they miss a deadline, they can just remove one of the sleep and call it a performance update =P
Changed up the UI for video players and removed the system UI (status and navigation bar) after Kitkat. This should make the main video player a bit more like the Youtube app design. However, the code for this is somewhat confusing, so please be sure to go over it. Though every time I have to deal with api levels, I feel a little more dead inside. Don't show overlay after resuming from buffering #1151 is also included as part of this commit.

…rceManager. -Added nonnull and final constraints to variables in MediaSourceManager. -Added nonnull and final constraints on context related objects in BasePlayer. -Fixed Hls livestreams crashing player when behind live window for too long. -Fixed cache miss when InfoCache key mismatch between StreamInfo and StreamInfoItem.

-Improved player sync calls to recognize different metadata updates. -Changed MediaSourceManager to synchronize only after timeline changes and reenabled multiple sync calls to player. -Renamed listener and synchronization methods related to MediaSourceManager.

-Reenabled full window loading in MediaSourceManager.

-[TeamNewPipe#1151] Hide video player UI on playing to avoid unnecessary interruptions after pause, seek and resize.

karyogamy · 2018-03-04T05:06:53Z

More updates:

Added seamless shuffling. Now the only sources of non-seamless playback is buffering, player exception and quality switching. Nothing we can do about the first two, and the third is probably non-trivial considering Improve resolution selection #1103. However, this is a major tech debt that we will have to address eventually, since the current quality selector implementation is still quite hacky.
Added fast inexact seeking. It might be syncing to the nearest keyframe and off by a few seconds, but boy is it fast. This is another feature that is available in ExoPlayer, which we can simply enable. A setting toggle has been added to let users try it out.

ghost · 2018-03-04T06:21:52Z

off by a few seconds

I personally hate players that do that, could you create a setting for that?

karyogamy · 2018-03-04T06:23:34Z

I did. It's disabled by default.

ghost · 2018-03-04T06:24:28Z

Oh, I didn't see that part, thanks!

… the last item on play queue is playing. -Added toggle to enable auto-queuing. -Modified main video player to only pause the video onPause. -Fixed main video player not saving play queue state onStop.

-Disabled auto queuing when repeating is enabled. -Added method to use startForegroundService instead of startService in sdk 26 and up.

karyogamy · 2018-03-06T19:04:36Z

app/src/main/java/org/schabi/newpipe/util/InfoCache.java

-            lruCache.put(keyOf(info), data);
+
+        final long expirationMillis;
+        if (info.getServiceId() == SoundCloud.getServiceId()) {


@theScrabi
I was hoping to receive some more review and discussions before having the PR merged, since there might be parts that would lead to tech debts.
For example, this part where different services can have different cache expiration. SoundCloud was added here since their API seems to fail sporadically and would cause the StreamInfo to have bad stream urls and need to be reloaded.

This probably should be part of the extractor, or at least ServiceHelper. Please consider removing this.

I see, I merged early, so now we have to take care about this in the aftermath.

For example, this part where different services can have different cache expiration. SoundCloud was added here since their API seems to fail sporadically and would cause the StreamInfo to have bad stream urls and need to be reloaded.

So you'd suggest a new field in StreamInfo that tells the URls expiration time? If so we should add this.

Please consider removing this.

You mean the part that checks for SoundCloud? Right thats a hack and needs a proper solution.

I'm currently working on a PR that lets user wipe the InfoCache, so we can remove this check. I've also reduce the default cache time to 1 hr instead of 4. I've also moved that check to ServiceHelper for now and reduce the cache time for SoundCloud to 5 minutes.

However, this is not a long term solution, as SoundCloud stream url seems to consistently expire after a few minutes if they are not played, at least from my side. We will need to figure out exactly why this is happening, since it's probably a bad idea just slapping on an arbitrary expiration time for each service.

Well we might have similare problems with YouTube as some issues point out. But you are right, we should investigate this.

karyogamy · 2018-03-06T19:06:47Z

app/src/main/java/org/schabi/newpipe/player/helper/PlayerHelper.java

@@ -83,6 +98,58 @@ public static String resizeTypeOf(@NonNull final Context context,
        }
    }

+    @NonNull
+    public static String cacheKeyOf(@NonNull final StreamInfo info, @NonNull VideoStream video) {
+        return info.getUrl() + video.getResolution() + video.getFormat().getName();


Just for the record. The cache key generation here might be too naive. Hopefully someone can find a better alternative, such as generating the VideoStream or AudioStream identifier in the extractor.

So you think we should distinguish between audio and video data as well?

Sure, I think each normal stream should have a unique id they can use regardless of their url, since it changes all the time.

Yes that makes better sense.
This is why the UrlIdHandlers exist. Maybe these should be used at this point.

oceanwaves90 · 2018-03-10T16:53:02Z

Hi! Thank you for your works!

By the way, at times the sync of live stream doesn't work. It plays a certain interval of the stream and so the sync button plays always from the same point of the stream. The live stream is this, which has been broadcasted for several days.

New playing also repeats the above problem. It starts from the very point of stream. It never goes to the present live streamed point because the playing ends at the point of 2 hours.

And the live-watching number isn't updated in subscription window and in the detail window, while the live-watching number of the browsed result of the stream is correct.

Restarting the app also repeats the problems.

Do the problems have to do with caching the stream data?

(+edit)
After I imported a backuped database file, it works well. But I'm not sure whether System.exit, which is called when imported, reset the cached data or the backuped database file replaced the cached data.

karyogamy · 2018-03-10T19:56:59Z

@oceanwaves90
You are right, this is mostly a problem with the cache. We are currently caching the data for the latest 60 Youtube page accesses for 4 hours. So if anything on a cache page updates in the real time, it's likely they aren't reflected. System.exit probably obliterated that cache. Also, I don't think live-watching number is even displayed right now, since this requires constant web requests to the stream url which is about 200kb each, and can add up quickly.

I also reproduced the sync button problem on the url you gave. I think this is probably due to the livestream being renewed on the uploader end, causing the livestream manifest to automatically transition into a normal stream, as I had no problem streaming this on live-edge for about 8 hours. So, unless this problem becomes rampant, I'd avoid writing extra logic to deal with this kind of cases, since we don't know when a stream url becomes stale.

However, now that we have more dynamic data with livestreaming, we should consider expiring Youtube stream page data sooner, as well as adding a quick button (on the top right dropdown) to let users wipe the cache manually.

oceanwaves90 · 2018-03-10T20:32:24Z

I see! That will be great! I'll look forward to the next updates!

karyogamy added 7 commits February 24, 2018 21:54

-Updated Exoplayer to 2.7.0.

8803b60

-PoC for new seamless stream loading mechanism.

-Fixed media source update index check.

19cbcd0

-Fixed media source manager excessive loading. -Remove unneeded fields in loaded media source.

-Fixed failed media source not treated as ready.

ac431e3

-Fixed potential NPE when obtaining broadcast receiver.

1444fe5

-Extracted expiration time in media source manager. -Re-enabled long click on live stream info items. -Fixed dash source building to use mpd instead of extractor.

-Added perpetual extractor source loading on network failures.

77da40e

-Fixed play queue playlist desynchronization caused by media source manager window loading expansion on sublist prior to current item. -Fixed failed media source not treated as ready for playback.

karyogamy mentioned this pull request Mar 1, 2018

Don't show overlay after resuming from buffering #1151

Closed

karyogamy added 3 commits February 28, 2018 17:45

-Added serialized cache for transferring serializable objects too lar…

a1220c7

…ge for intent transactions. -Fixed potential transaction too large exceptions for player intents.

-Re-added loading for items prior to current index in MediaSourceMana…

9ea08c8

…ger to allow faster access time. -Added some null checks annotation.

karyogamy added 4 commits March 3, 2018 11:42

-Added seamless shuffling.

59558ef

-Reenabled full window loading in MediaSourceManager.

-Removed system ui on main player for Kitkat or above.

7f068b6

-[TeamNewPipe#1151] Hide video player UI on playing to avoid unnecessary interruptions after pause, seek and resize.

karyogamy force-pushed the exoplayer-update branch from 49c620b to 7f068b6 Compare March 4, 2018 05:06

-Added auto-queuing to allow next or related streams to queue up when…

d01aeab

… the last item on play queue is playing. -Added toggle to enable auto-queuing. -Modified main video player to only pause the video onPause. -Fixed main video player not saving play queue state onStop.

karyogamy mentioned this pull request Mar 5, 2018

Autoplay next video #165

Closed

-Fixed main video player losing state when killed in background.

b34160e

-Disabled auto queuing when repeating is enabled. -Added method to use startForegroundService instead of startService in sdk 26 and up.

karyogamy changed the title ~~[WIP] Live Streaming and ExoPlayer Shenanigans~~ Live Streaming and ExoPlayer Shenanigans Mar 6, 2018

theScrabi merged commit b34160e into TeamNewPipe:dev Mar 6, 2018

karyogamy commented Mar 6, 2018

View reviewed changes

TobiGr mentioned this pull request Mar 6, 2018

Crash when opening video from start page #1149

Closed

This was referenced Mar 13, 2018

Youtube live not supported #195

Closed

Small things to do next #729

Closed

mauriciocolli mentioned this pull request Apr 7, 2018

Add setting to display hardware buttons while watching a video #1283

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Live Streaming and ExoPlayer Shenanigans #1147

Live Streaming and ExoPlayer Shenanigans #1147

karyogamy commented Feb 26, 2018 •

edited

karyogamy commented Mar 1, 2018 •

edited

karyogamy commented Mar 4, 2018

ghost commented Mar 4, 2018 via email

karyogamy commented Mar 4, 2018 •

edited

ghost commented Mar 4, 2018 via email

karyogamy Mar 6, 2018

theScrabi Mar 11, 2018 •

edited

karyogamy Mar 11, 2018

theScrabi Mar 12, 2018

karyogamy Mar 6, 2018

theScrabi Mar 11, 2018

karyogamy Mar 11, 2018

theScrabi Mar 12, 2018

oceanwaves90 commented Mar 10, 2018 •

edited

karyogamy commented Mar 10, 2018 •

edited

oceanwaves90 commented Mar 10, 2018

Live Streaming and ExoPlayer Shenanigans #1147

Live Streaming and ExoPlayer Shenanigans #1147

Conversation

karyogamy commented Feb 26, 2018 • edited

karyogamy commented Mar 1, 2018 • edited

karyogamy commented Mar 4, 2018

ghost commented Mar 4, 2018 via email

karyogamy commented Mar 4, 2018 • edited

ghost commented Mar 4, 2018 via email

karyogamy Mar 6, 2018

Choose a reason for hiding this comment

theScrabi Mar 11, 2018 • edited

Choose a reason for hiding this comment

karyogamy Mar 11, 2018

Choose a reason for hiding this comment

theScrabi Mar 12, 2018

Choose a reason for hiding this comment

karyogamy Mar 6, 2018

Choose a reason for hiding this comment

theScrabi Mar 11, 2018

Choose a reason for hiding this comment

karyogamy Mar 11, 2018

Choose a reason for hiding this comment

theScrabi Mar 12, 2018

Choose a reason for hiding this comment

oceanwaves90 commented Mar 10, 2018 • edited

karyogamy commented Mar 10, 2018 • edited

oceanwaves90 commented Mar 10, 2018

karyogamy commented Feb 26, 2018 •

edited

karyogamy commented Mar 1, 2018 •

edited

karyogamy commented Mar 4, 2018 •

edited

theScrabi Mar 11, 2018 •

edited

oceanwaves90 commented Mar 10, 2018 •

edited

karyogamy commented Mar 10, 2018 •

edited