Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid bandwith throttling #6923

Open
error39 opened this issue Sep 22, 2015 · 18 comments
Open

Avoid bandwith throttling #6923

error39 opened this issue Sep 22, 2015 · 18 comments
Labels
request

Comments

@error39
Copy link

@error39 error39 commented Sep 22, 2015

Hola Ricardo,

I Wondered if there's a way of implementing an option for circumventing a specific kind of bandwidth throtling?

Let me explain: I noticed quite a few providers, for example soundcloud and mixcloud, will serve the first few megabytes of a stream at maximum speed, but after this the speed drops just to the amount needed for streaming the served media at its bitrate.
Like this it takes a long time to download certain media with youtube-dl. How I used to circumvent this is by stopping & resuming the download for example 20 times for a 100mb file, in case they serve the first 5mb unthrottled.

Technically this works very well, but it's not very handy in a practical way. So I created my little helper script where I use youtube-dl to generate the download-url, and let Curl download the actual file, by downloading the first XXkb block, than the second XXkb block, etc... using the range header start & end values, and a while loop.

This works out great for me, but I wonder if it would be very difficult to implement this in the downloader classes of youtube-dl? With an optional switch to activate this 'chunk-downloading', or even the ability for contributors to implement it in the extractors - if the specific service is known for using these 'first part fast' throttling mechanisms - would really benefit the download speed for a lot of users, in my humble opinion :)

So, my questions are: Do you think it's a good idea to implement this kind of functionality and how difficult will it be to do so? If you consider implementing this in the future, I like to help in any way I can
to make this possible.

Thanks for creating this great 'little' tool, I use it regularly and am very happy it exists!

Saludos,

Erik
@rg3
Copy link
Collaborator

@rg3 rg3 commented Sep 22, 2015

Thanks for your kind words on behalf of the current youtube-dl maintainers. As for the feature request, wait for their reply.

Best regards.
@error39
Copy link
Author

@error39 error39 commented Sep 26, 2015

I would like to look into this myself, as there's only a limited amount of time maintainers have to look into feature requests like this, I guess. Anyone who can give me a headstart with some direction to which files/functions are actually responsible for handling the actual download of the video or audio file?
One more question; Is there a guide/documentation on how to compile the python code yourself on a linux system?

Thanks!

edit: Found the developer notes myself: https://github.com/rg3/youtube-dl/#developer-instructions
@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Sep 26, 2015

A good suggestion. For most cases (including soundcloud), downloading happens in downloader/http.py.
@yan12125 yan12125 added the request label Sep 26, 2015
@jaimeMF
Copy link
Collaborator

@jaimeMF jaimeMF commented Sep 26, 2015

Note that until it gets implemented in the http downloader you may want to try aria2 with --external-downloader aria2c, it uses multiple connections which could help in your case.
@error39
Copy link
Author

@error39 error39 commented Sep 26, 2015

@yan12125 Thanks for your help, I will try to see if/how it's possible to implement this kind of functionality in http.py (I'm a python n00b, but it's always fun learning.. :) )

@jaimeMF It speeds up my download, but it opens multiple connections at the same time, so it's not exactly what I'm after (It increases the server load, which is one reason download accelerators are not so popular among server admins, and chances are they'll try to block those methods if possible) But nevertheless thanks a lot for the valuable tip! It might help others looking for a way to make downloads happen faster.
@Degru
Copy link

@Degru Degru commented Nov 15, 2016

Something like this would be great! Perhaps an option so that if speed drops below a certain rate, re-start the download from the current point. This happens for me on Youtube. I download on a 400mbps connection and get 3MB/s downloads at times for some videos after the first couple hundred megabytes.
@Serkora Serkora mentioned this issue Dec 18, 2016
Open
5 of 8 tasks complete
@yan12125 yan12125 mentioned this issue Mar 25, 2017
Closed
@yan12125 yan12125 mentioned this issue Apr 19, 2017
Closed
4 of 8 tasks complete
@ghost
Copy link

@ghost ghost commented Jan 16, 2018 •
edited by ghost

Hi. Seems to me that youtube just started doing this! Noticed today, wasn't present like 2 days ago.

First burst is at full speed (ie. 9.25MiB, not Mbps) then there are 2 bursts of 1.04 MBit/s each lasting 1 sec happening every 2.5 sec (equivalent of an average of 400Kbps/sec) - EDIT: according to nload.

EDIT2: first burst gets 0.5MiB of the file (or, 1.1% of a 45.78MiB file) - I'm assuming this is true for any file, although this particular one was only the audio part (a .webm) of a video.

Before today, it used to download at full speed constantly (ie. 100Mbps/sec).

EDIT:3: looks like if I restart it 5 times (ie. start youtube-dl, C-c after like 7 sec(ie. let it run just as long as the first half meg gets downloaded at full speed then C-c), repeat 5 times) it unlocks the full speed the 6th time. (same file above got transferred in 1 sec from 75% to 100% after this happened).

EDIT4: I've just retested edit3 on 3 new videos(but only the audio part, because that's what I needed) and there was only one repeat needed: ie. first try got stalled after first burst, I C-c, rerun youtube-dl, then it downloads the remainder at full speed.

here's full output for edit4 
$ downaudio https://www.youtube.com/watch?v=1PtC7D9euHU
[youtube] 1PtC7D9euHU: Downloading webpage
!! host= www.youtube.com
[youtube] 1PtC7D9euHU: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] 1PtC7D9euHU: Extracting video information
!! host= r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a
[download]   6.7% of 7.43MiB at  9.70MiB/s ETA 00:00^C
ERROR: Interrupted by user

real	0m6.358s
user	0m3.523s
sys	0m0.163s
$ downaudio https://www.youtube.com/watch?v=1PtC7D9euHU
[youtube] 1PtC7D9euHU: Downloading webpage
!! host= www.youtube.com
[youtube] 1PtC7D9euHU: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] 1PtC7D9euHU: Extracting video information
[download] Resuming download at byte 523264
!! host= r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a
[download] 100% of 7.43MiB in 00:00
[ffmpeg] Correcting container in "As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a"
[ffmpeg] Post-process file As Predicted the Neoliberal Media is Trying to Get the Dems to Seal a DACA Deal-1PtC7D9euHU.m4a exists, skipping

real	0m5.602s
user	0m4.045s
sys	0m0.363s
$ downaudio https://www.youtube.com/watch?v=trH6Uw_Rs-E
[youtube] trH6Uw_Rs-E: Downloading webpage
!! host= www.youtube.com
[youtube] trH6Uw_Rs-E: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] trH6Uw_Rs-E: Extracting video information
!! host= r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a
[download]   5.5% of 9.09MiB at  9.71MiB/s ETA 00:00^C
ERROR: Interrupted by user

real	0m6.187s
user	0m3.467s
sys	0m0.157s
$ downaudio https://www.youtube.com/watch?v=trH6Uw_Rs-E
[youtube] trH6Uw_Rs-E: Downloading webpage
!! host= www.youtube.com
[youtube] trH6Uw_Rs-E: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] trH6Uw_Rs-E: Extracting video information
[download] Resuming download at byte 523264
!! host= r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a
[download] 100% of 9.09MiB in 00:00
[ffmpeg] Correcting container in "Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a"
[ffmpeg] Post-process file Project Veritas Exposes Twitter Corruption - Twitter Denies It (Use Alt Tech!)-trH6Uw_Rs-E.m4a exists, skipping

real	0m5.766s
user	0m4.054s
sys	0m0.411s
$ downaudio https://www.youtube.com/watch?v=nKxjG9F0iw8
[youtube] nKxjG9F0iw8: Downloading webpage
!! host= www.youtube.com
[youtube] nKxjG9F0iw8: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] nKxjG9F0iw8: Extracting video information
!! host= r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
[download] Destination: VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a
[download]   6.0% of 8.27MiB at  9.67MiB/s ETA 00:00^C
ERROR: Interrupted by user

real	0m6.694s
user	0m3.474s
sys	0m0.183s
$ downaudio https://www.youtube.com/watch?v=nKxjG9F0iw8
[youtube] nKxjG9F0iw8: Downloading webpage
!! host= www.youtube.com
[youtube] nKxjG9F0iw8: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] nKxjG9F0iw8: Extracting video information
[download] Resuming download at byte 523264
!! host= r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
[download] Destination: VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a
[download] 100% of 8.27MiB in 00:00
[ffmpeg] Correcting container in "VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a"
[ffmpeg] Post-process file VR to ThuleanPerspective Re His VR on Adsense-nKxjG9F0iw8.m4a exists, skipping

real	0m5.734s
user	0m4.097s
sys	0m0.357s
$ type downaudio
downaudio is aliased to `time youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio'
$ cat /etc/hosts|grep r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.210 r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
$ dig r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
r7.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.210 r7---sn-h5bt5guxax3vh-c0qy.googlevideo.com
$ cat /etc/hosts|grep r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
145.236.72.15 r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
$ dig r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
r4.sn-h5bt5guxax3vh-c0q6.googlevideo.com. r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
145.236.72.15 r4---sn-h5bt5guxax3vh-c0q6.googlevideo.com
$ cat /etc/hosts|grep r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
145.236.72.83 r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
$ dig r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
r8.sn-h5bt5guxax3vh-c0qd.googlevideo.com. r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com
145.236.72.83 r8---sn-h5bt5guxax3vh-c0qd.googlevideo.com

EDIT5: Just figured out that --socket-timeout 1 works flawlessly, especially since I've encountered this video* which wouldn't decide to unlock full speed even after like 20 restarts! Thank you for this option, btw! Amazing!
* I say video but I'm only downloading the audio part.
EDIT6: Also need --retries infinite or it will end after the default of 10.
However, there is a jump from 60.8% to 100% even though the output seems to be of correct size:
-rw-r--r-- 1 xftroxgpx xftroxgpx 28768063 08.01.2018 11:18 'VERY STABLE GENIUS-u-yLGFuu2dc.opus'
I'll take that to mean that after 34 retries it unlocked full speed.

actual console output 
$ downaudio --socket-timeout 1 --retries infinite https://www.youtube.com/watch?v=u-yLGFuu2dc
[youtube] u-yLGFuu2dc: Downloading webpage
!! host= www.youtube.com
[youtube] u-yLGFuu2dc: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] u-yLGFuu2dc: Extracting video information
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]   1.8% of 27.89MiB at  9.79MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of inf)...
[download] Resuming download at byte 523264
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]   3.6% of 27.89MiB at  9.92MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 2 of inf)...
[download] Resuming download at byte 1046528
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]   5.4% of 27.89MiB at  9.89MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of inf)...
[download] Resuming download at byte 1569792
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]   7.2% of 27.89MiB at  9.86MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 4 of inf)...
[download] Resuming download at byte 2093056
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]   8.9% of 27.89MiB at  9.84MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 5 of inf)...
[download] Resuming download at byte 2616320
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  10.7% of 27.89MiB at  9.90MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 6 of inf)...
[download] Resuming download at byte 3139584
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  12.5% of 27.89MiB at  9.79MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 7 of inf)...
[download] Resuming download at byte 3662848
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  14.3% of 27.89MiB at  9.61MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 8 of inf)...
[download] Resuming download at byte 4186112
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  16.1% of 27.89MiB at  9.73MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 9 of inf)...
[download] Resuming download at byte 4709376
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  17.9% of 27.89MiB at  9.85MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 10 of inf)...
[download] Resuming download at byte 5232640
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  19.7% of 27.89MiB at  9.98MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 11 of inf)...
[download] Resuming download at byte 5755904
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  21.5% of 27.89MiB at  9.87MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 12 of inf)...
[download] Resuming download at byte 6279168
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  23.3% of 27.89MiB at  9.67MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 13 of inf)...
[download] Resuming download at byte 6802432
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  25.1% of 27.89MiB at  9.83MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 14 of inf)...
[download] Resuming download at byte 7325696
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  26.8% of 27.89MiB at  9.91MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 15 of inf)...
[download] Resuming download at byte 7848960
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  28.6% of 27.89MiB at  9.77MiB/s ETA 00:02[download] Got server HTTP error: The read operation timed out. Retrying (attempt 16 of inf)...
[download] Resuming download at byte 8372224
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  30.4% of 27.89MiB at  9.89MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 17 of inf)...
[download] Resuming download at byte 8895488
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  32.2% of 27.89MiB at  9.86MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 18 of inf)...
[download] Resuming download at byte 9418752
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  34.0% of 27.89MiB at  9.87MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 19 of inf)...
[download] Resuming download at byte 9942016
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  35.8% of 27.89MiB at  9.97MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 20 of inf)...
[download] Resuming download at byte 10465280
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  37.6% of 27.89MiB at  9.81MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 21 of inf)...
[download] Resuming download at byte 10988544
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  39.4% of 27.89MiB at  9.88MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 22 of inf)...
[download] Resuming download at byte 11511808
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  41.2% of 27.89MiB at  9.85MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 23 of inf)...
[download] Resuming download at byte 12035072
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  42.9% of 27.89MiB at  9.84MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 24 of inf)...
[download] Resuming download at byte 12558336
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  44.7% of 27.89MiB at  9.75MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 25 of inf)...
[download] Resuming download at byte 13081600
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  46.5% of 27.89MiB at  9.93MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 26 of inf)...
[download] Resuming download at byte 13604864
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  48.3% of 27.89MiB at  9.93MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 27 of inf)...
[download] Resuming download at byte 14128128
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  50.1% of 27.89MiB at  9.80MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 28 of inf)...
[download] Resuming download at byte 14651392
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  51.9% of 27.89MiB at  9.64MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 29 of inf)...
[download] Resuming download at byte 15174656
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  53.7% of 27.89MiB at  9.78MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 30 of inf)...
[download] Resuming download at byte 15697920
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  55.5% of 27.89MiB at  9.90MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 31 of inf)...
[download] Resuming download at byte 16221184
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  57.3% of 27.89MiB at  9.87MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 32 of inf)...
[download] Resuming download at byte 16744448
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  59.1% of 27.89MiB at  9.76MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 33 of inf)...
[download] Resuming download at byte 17267712
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download]  60.8% of 27.89MiB at  9.88MiB/s ETA 00:01[download] Got server HTTP error: The read operation timed out. Retrying (attempt 34 of inf)...
[download] Resuming download at byte 17790976
!! host= r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
[download] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.webm
[download] 100% of 27.89MiB in 00:00
[ffmpeg] Destination: VERY STABLE GENIUS-u-yLGFuu2dc.opus
Deleting original file VERY STABLE GENIUS-u-yLGFuu2dc.webm (pass -k to keep)

real	0m44.638s
user	0m6.094s
sys	0m1.016s
$ type downaudio
downaudio is aliased to `time youtube-dl --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio'
$ cat /etc/hosts|grep r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
145.236.72.12 r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
$ dig r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
r1.sn-h5bt5guxax3vh-c0q6.googlevideo.com. r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
145.236.72.12 r1---sn-h5bt5guxax3vh-c0q6.googlevideo.com
$ type dig
dig is hashed (/home/xftroxgpx/bin/dig)
Note that downloading the video itself, is not as bad: seems to go fullspeed for the first few seconds, then limited to 20Mbit/sec). It's the audio which gets stalled bad.
This was referenced Jan 17, 2018
Closed
Closed
@ghost
Copy link

@ghost ghost commented Jan 18, 2018 •
edited by ghost

Partial workaround for automatically bypassing stalls while downloading the audio part from youtube, just add args: --socket-timeout 1 --retries infinite to youtube-dl (as per my edit5&6 above)

EDIT1: I guess it doesn't always work (or they were quick to update against the workaround above? xD EDIT3: actually, I can only surmise that different *.googlevideo.com servers have differing behaviours, also see my edits in my prev. post) 
$ downaudio https://www.youtube.com/watch?v=rPEU2KJwY6Q
[youtube] rPEU2KJwY6Q: Downloading webpage
!! host= www.youtube.com
[youtube] rPEU2KJwY6Q: Downloading video info webpage
!! host= www.youtube.com
!! host= www.youtube.com
[youtube] rPEU2KJwY6Q: Extracting video information
!! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
!! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm
[download]   1.0% of 50.51MiB at  9.87MiB/s ETA 00:05[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of inf)...
[download] Resuming download at byte 523264
!! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
!! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm
[download]   2.0% of 50.51MiB at  9.99MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 2 of inf)...
[download] Resuming download at byte 1046528
!! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
!! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm
[download]   3.0% of 50.51MiB at  9.95MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of inf)...
[download] Resuming download at byte 1569792
!! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
!! host= r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
[download] Destination: Hbomberguy - An Unmeasured Response-rPEU2KJwY6Q.webm
[download]   4.0% of 50.51MiB at  9.87MiB/s ETA 00:04[download] Got server HTTP error: The read operation timed out. Retrying (attempt 4 of inf)...
[download] Resuming download at byte 2093056
!! host= r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
ERROR: unable to download video data: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

real	0m9.234s
user	0m3.877s
sys	0m0.223s
$ type downaudio
downaudio is aliased to `time youtube-dl --socket-timeout 1 --retries infinite --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" --extract-audio'
$ youtube-dl --version
2018.01.14
$ cat /etc/hosts|grep r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.208 r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
$ dig r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
r5.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.208 r5---sn-h5bt5guxax3vh-c0qy.googlevideo.com
$ cat /etc/hosts|grep r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.205 r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
$ dig r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
r2.sn-h5bt5guxax3vh-c0qy.googlevideo.com. r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com
31.46.22.205 r2---sn-h5bt5guxax3vh-c0qy.googlevideo.com

EDIT2: some bash workaround for edit1: $ (exit 1) || while test "$?" -ne "0"; do downaudio https://www.youtube.com/watch?v=rPEU2KJwY6Q; done

EDIT4: alright, I've decided to stop using youtube altogether, replacing it with reading books.
@wltprgm
Copy link

@wltprgm wltprgm commented Jan 18, 2018

This is not a workaround but my own findings.

I downloaded NewPipe from F-droid and standalone apk Snaptube to test on this slow downloading issue on m4a and dash video 1080p.

Snaptude speed is like youtube-dl, slow.

But Newpipe downloading speed is superb. Not sure why, I couldn't really read any code now as a newbie in programming.
skeeto added a commit to skeeto/.emacs.d that referenced this issue Jan 18, 2018
@skeeto
Work around youtube-dl / YouTube throttling issue
0a226fd 
I have no idea what's going on, but this completely fixes my problem
at the moment. Otherwise it takes an hour to download a single video.
See also:

* ytdl-org/youtube-dl#15275
* ytdl-org/youtube-dl#6923
* ytdl-org/youtube-dl#11476
* https://www.reddit.com/r/DataHoarder/comments/7qpuhn/youtubedl_being_throttled/
@zouhair
Copy link
Contributor

@zouhair zouhair commented Jan 20, 2018

It seems they are systematically throttling all DASH streams, I doubt there can be a solution client side.
@norcaljohnny
Copy link

@norcaljohnny norcaljohnny commented Jan 20, 2018

Just something worth mentioning. If you load a video or audio directly into an html5 player, then download is instant. I am considering quickly creating a php page/js form to handle the dl and conversion of mp3. I believe it is faster than some other options I was left with.
@zouhair
Copy link
Contributor

@zouhair zouhair commented Jan 20, 2018

Even for DASH 1080p+ ?
@coolguruji
Copy link

@coolguruji coolguruji commented Jan 20, 2018

First i thought my server provider is throttling my bandwidth speed but they told me - "No, we are not"
Then i tried with multiple speedtest via command line and i saw that download speed is awesome of my server.
Now i started to google about it on web. As i can say know,
Youtube.com maybe throttling our requests and slowing it down. I download webm/m4a format and download speed is 80 to 250 kbps right now, however it should be 20 to 40 Mbps. using aria2c is not giving much boost.

Now the question is how can we fix it?
Please suggest any idea!
@zouhair
Copy link
Contributor

@zouhair zouhair commented Jan 20, 2018

As far as I understand it (don't take my word for it) this is a server side problem. Youtube seems to be throttling everything but the "best" stream. As far as I know, there is not much we can do on the client side, and I hope to be wrong.
@ghost
Copy link

@ghost ghost commented Jan 20, 2018

The download on Youtube is now very annoying. But I have an idea how to handle this. How about if youtube-dl includes a list of VPN servers that you would define yourself in a config file for the download?
@zouhair
Copy link
Contributor

@zouhair zouhair commented Jan 20, 2018

VPN won't solve a thing, this is a server side thing.
@ghost
Copy link

@ghost ghost commented Jan 20, 2018

Sure it is a server problem. The idea is that youtube-dl fooled the server, the download would be downloaded by different clients, depending on the number of VPN servers that you defined in the config list
@porjo
Copy link

@porjo porjo commented Jan 21, 2018

I'm seeing the same throttling with audio-only downloads (-f worstaudio) in the last week. Downloads that previously would take seconds now take upwards of several minutes, regardless of what Internet connection I use. As an earlier commentor suggested, using --external-downloader aria2c appears to work around this.
@ytdl-org ytdl-org deleted a comment from kik4444 Jan 22, 2018
@ytdl-org ytdl-org deleted a comment from Phil57 Jan 22, 2018
@ytdl-org ytdl-org locked as off topic and limited conversation to collaborators Jan 22, 2018
@ytdl-org ytdl-org deleted a comment from siwyd Jan 22, 2018
@ytdl-org ytdl-org deleted a comment from ArcAster Jan 22, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
request
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
@rg3 @zouhair @jaimeMF @porjo @yan12125 @Degru @error39 @norcaljohnny @wltprgm @coolguruji
You can’t perform that action at this time.