Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow download rate: Youtube throttling #11

Closed
glubsy opened this issue Jul 14, 2021 · 9 comments
Closed

Slow download rate: Youtube throttling #11

glubsy opened this issue Jul 14, 2021 · 9 comments

Comments

@glubsy
Copy link
Owner

glubsy commented Jul 14, 2021

Youtube throttles downloads from URL endpoints not signed with some kind of cryptographic token.

Pytube implemented a fix (pytube linked issue),

invidious implemented a fix,

yt-dlp issue about this topic.

Possible workaround but needs more testing: having the javascript youtube player active in a web browser (ie. the video page open, but the video itself probably does not need to be playing).

Possible solutions: currently attempting to merge some of Pytube methods in the hope to get a signed URL for every stream. Will it need to be refreshed? And how frequently? No idea yet.

Also looking into integrating yt-dlp and streamlink somehow but it will take time.
@glubsy
Copy link
Owner Author

glubsy commented Jul 22, 2021

This should be fixed after half integrating pytube and using their way of computing the signed url token, with commit 67201e6

@glubsy glubsy closed this as completed Jul 22, 2021
@glubsy glubsy reopened this Aug 28, 2021
@glubsy
Copy link
Owner Author

glubsy commented Sep 17, 2021 •
edited
Loading

This is not entirely fixed yet. Some live streams appear to be throttled while others are not. Perhaps the signature needs to be applied periodically, which would force us to refresh the download URLs for each stream every now and then. If that is the case, we'll need to figure out that time period somehow.

It has been reported to me that restarting the download process brings back normal download speed to normal for a little while, which corroborates this assumption.

@glubsy glubsy mentioned this issue Oct 13, 2021
Open
@snipergod87
Copy link

snipergod87 commented Oct 13, 2021 •
edited
Loading

It may indeed be throttling, There was a public livestream earlier today that ended 3 hours ago (started 4h ago).
LiveStreamSaver is still downloading segments
2021-10-13 11:30:24,825 - INFO - livestream_saver - Found live: UWrIOOoFIaM title: 【Moona Boona】MysStory【Episode 2】. Downloading... 2021-10-13 11:30:24,826 - INFO - download.UWrIOOoFIaM - Will start downloading from segment number 0. 2021-10-13 11:30:25,329 - INFO - download.UWrIOOoFIaM - Stream seems to be viewed live. Good. 2021-10-13 11:30:25,330 - INFO - download.UWrIOOoFIaM - Stream status Status.OK Finding throttling function name finished regex search, matched: %s a\.[A-Z]&&\(b=a\.get\("n"\)\)&&\(b=([^(]+)\(b\) Finding throttling function name finished regex search, matched: %s a\.[A-Z]&&\(b=a\.get\("n"\)\)&&\(b=([^(]+)\(b\) 2021-10-13 11:30:25,878 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="299" mime_type="video/mp4" res="1080p" fps="60fps" vcodec="avc1.4d402a" progressive="False" type="video"> 2021-10-13 11:30:25,879 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="136" mime_type="video/mp4" res="720p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video"> 2021-10-13 11:30:25,879 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="298" mime_type="video/mp4" res="720p" fps="60fps" vcodec="avc1.4d4020" progressive="False" type="video"> 2021-10-13 11:30:25,880 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="135" mime_type="video/mp4" res="480p" fps="30fps" vcodec="avc1.4d401f" progressive="False" type="video"> 2021-10-13 11:30:25,880 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="134" mime_type="video/mp4" res="360p" fps="30fps" vcodec="avc1.4d401e" progressive="False" type="video"> 2021-10-13 11:30:25,880 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="133" mime_type="video/mp4" res="240p" fps="30fps" vcodec="avc1.4d4015" progressive="False" type="video"> 2021-10-13 11:30:25,881 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="160" mime_type="video/mp4" res="144p" fps="30fps" vcodec="avc1.4d400c" progressive="False" type="video"> 2021-10-13 11:30:25,881 - INFO - download.UWrIOOoFIaM - Available <Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio"> 2021-10-13 11:30:25,882 - INFO - download.UWrIOOoFIaM - Selected video <Stream: itag="299" mime_type="video/mp4" res="1080p" fps="60fps" vcodec="avc1.4d402a" progressive="False" type="video"> 2021-10-13 11:30:25,882 - INFO - download.UWrIOOoFIaM - selected audio <Stream: itag="140" mime_type="audio/mp4" abr="128kbps" acodec="mp4a.40.2" progressive="False" type="audio"> Downloading segment 1436...
It is currently 15:50

Edit: at 17:30 critical error, Error 403 Forbidden.
It downloaded 30 minutes of a 1h stream.

Edit 2: I am being throttling with youtube-dl to 60KB's so its most likely the same for livestream-saver. Currently to download an uploaded video I am using aria2c through youtube-dl with a few arguments -j 16 -x 16 -s 16 at least I can get around 1MB/s but I dont think aria2c can be incorporated here.

@glubsy
Copy link
Owner Author

glubsy commented Oct 14, 2021

I am being throttling with youtube-dl to 60KB

Yup, I have been experiencing the exact same issue. I don't have a clear solution yet. It's something to investigate further.

@undulum
Copy link

undulum commented Nov 12, 2021

In ytarchive, this was fixed by using the Android player response, the same strategy that yt-dlp used:
https://github.com/Kethsar/ytarchive/compare/50043024..d0380487

@glubsy
Copy link
Owner Author

glubsy commented Nov 12, 2021

@undulum thanks for that link. I will have to work on this and implement the same solution.
I just hope this is the actual definitive solution and that it will last.

@glubsy glubsy mentioned this issue Dec 3, 2021
Closed
glubsy added a commit that referenced this issue Dec 14, 2021
@glubsy
Make API v1 requests
dcb62de 
* Fetch additional metadata for each video in monitor mode through the
  API. This is needed to get a video description in order to compare regex
against it and determine whether or not the hook command should trigger.
* Get download URL via the ANDROID client parameters to get a pre-signed
  URL and get around the throttling issue #11.
@glubsy
Copy link
Owner Author

glubsy commented Dec 16, 2021

The issue should be fixed (for now) in commit dcb62de.

@glubsy
Copy link
Owner Author

glubsy commented Dec 24, 2021 •
edited
Loading

Just a heads up, the current implementation is incomplete, so members-only videos are not accessible, even with cookies loaded. A workaround is to spawn yt-dlp as hook command (and use the --skip-download flag or skip_download = true in the config file).

Remember to use --live-from-start with the latest version of yt-dlp (it can download from the first segment).

@glubsy
Copy link
Owner Author

glubsy commented Jan 10, 2022

Members-only videos should work now, as of d0a06d1.

@glubsy glubsy closed this as completed Jan 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@snipergod87 @glubsy @undulum