Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow 'Downloading webpage' & 'Video info' steps, stalling downloading from youtube #18935

Closed
diziet opened this issue Jan 20, 2019 · 1 comment
Closed

Slow 'Downloading webpage' & 'Video info' steps, stalling downloading from youtube #18935

diziet opened this issue Jan 20, 2019 · 1 comment
Labels
cant-reproduce

Comments

@diziet
Copy link

@diziet diziet commented Jan 20, 2019

  • I've verified and I assure that I'm running youtube-dl 2019.01.17
$ youtube-dl --version
2019.01.17

Before submitting an issue make sure you have:

  • At least skimmed through the README, most notably the FAQ and BUGS sections
  • Searched the bugtracker for similar issues including closed ones
  • Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

image

What is the purpose of your issue?

  • Bug report (encountered problems with youtube-dl)
$ youtube-dl -v 'https://www.youtube.com/watch?v=DRRttXwx5yY'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=DRRttXwx5yY']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.01.17
[debug] Python version 2.7.15 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: avconv 12.3, avprobe 12.3, ffmpeg 4.1, ffprobe 4.1, phantomjs 2.1.1
[debug] Proxy map: {}
[youtube] DRRttXwx5yY: Downloading webpage
[youtube] DRRttXwx5yY: Downloading video info webpage
[youtube] {18} signature length 42.44, html5 player vfl-jbnrr
[youtube] {36} signature length 42.44, html5 player vfl-jbnrr
[youtube] {17} signature length 42.44, html5 player vfl-jbnrr
[youtube] {135} signature length 42.44, html5 player vfl-jbnrr
[youtube] {244} signature length 42.44, html5 player vfl-jbnrr
[youtube] {134} signature length 42.44, html5 player vfl-jbnrr
[youtube] {243} signature length 42.44, html5 player vfl-jbnrr
[youtube] {133} signature length 42.44, html5 player vfl-jbnrr
[youtube] {242} signature length 42.44, html5 player vfl-jbnrr
[youtube] {160} signature length 42.44, html5 player vfl-jbnrr
[youtube] {278} signature length 42.44, html5 player vfl-jbnrr
[youtube] {140} signature length 42.44, html5 player vfl-jbnrr
[youtube] {171} signature length 42.44, html5 player vfl-jbnrr
[youtube] {249} signature length 42.44, html5 player vfl-jbnrr
[youtube] {250} signature length 42.44, html5 player vfl-jbnrr
[youtube] {251} signature length 42.44, html5 player vfl-jbnrr
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'https://r1---sn-fxc25nn-nwje.googlevideo.com/videoplayback?expire=1548043017&lmt=1517572649119080&ei=qe5EXJ_MCsGDkwa3kqjoBA&c=WEB&initcwndbps=3587500&id=o-ADBVzVH9rSvVtoN_Kr8HWhp5qNMWXFf2FEMPpw6KcY3Z&pl=16&dur=3899.520&fvip=1&mime=video%2Fwebm&ms=au%2Crdu&source=youtube&mv=m&mm=31%2C29&ipbits=0&mn=sn-fxc25nn-nwje%2Csn-n4v7knls&itag=244&ip=136.24.56.69&keepalive=yes&mt=1548021316&aitags=133%2C134%2C135%2C160%2C242%2C243%2C244%2C278&clen=49364993&key=yt6&requiressl=yes&gir=yes&sparams=aitags%2Cclen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&signature=53D2D8CFCFD231DFED7D893BD6F420B5C2453AB7.4D37E59CFFDB416C56F0D05313AD950E70EDE237&ratebypass=yes'
[download] Destination: Эпидемия Эльфийская рукопись-DRRttXwx5yY.f244.webm
[download] 100% of 47.08MiB in 06:19
[debug] Invoking downloader on u'https://r1---sn-fxc25nn-nwje.googlevideo.com/videoplayback?expire=1548043017&lmt=1542933845092300&ei=qe5EXJ_MCsGDkwa3kqjoBA&c=WEB&initcwndbps=3587500&id=o-ADBVzVH9rSvVtoN_Kr8HWhp5qNMWXFf2FEMPpw6KcY3Z&pl=16&dur=3899.541&fvip=1&mime=audio%2Fwebm&ms=au%2Crdu&source=youtube&mv=m&mm=31%2C29&ipbits=0&mn=sn-fxc25nn-nwje%2Csn-n4v7knls&itag=251&ip=136.24.56.69&keepalive=yes&mt=1548021316&txp=5411222&clen=62742082&key=yt6&requiressl=yes&gir=yes&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cexpire&signature=B7ACA7A2F53EADD7EF427190499EEF4F1605A5FF.90D7195D3C7B4C6DD73F5B1D4A0AE09775951877&ratebypass=yes'
[download] Destination: Эпидемия Эльфийская рукопись-DRRttXwx5yY.f251.webm
[download]  15.9% of 59.84MiB at 14.98MiB/s ETA 00:03

^^^ stuck on that for hours

and with --print-traffic

$ youtube-dl --print-traffic  https://www.youtube.com/watch?v=DRRttXwx5yY
[youtube] DRRttXwx5yY: Downloading webpage
send: u'GET /watch?has_verified=1&bpctr=9999999999&hl=en&v=DRRttXwx5yY&disable_polymer=true&gl=US HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=utf-8
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: Cache-Control: no-cache
header: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
header: X-Content-Type-Options: nosniff
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=31536000
header: Date: Sun, 20 Jan 2019 22:43:53 GMT
header: Server: YouTube Frontend Proxy
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Set-Cookie: VISITOR_INFO1_LIVE=YE9rggcKOH4; path=/; domain=.youtube.com; expires=Fri, 19-Jul-2019 22:43:53 GMT; httponly
header: Set-Cookie: GPS=1; path=/; domain=.youtube.com; expires=Sun, 20-Jan-2019 23:13:53 GMT
header: Set-Cookie: YSC=Z-mDXy9OhLY; path=/; domain=.youtube.com; httponly
header: Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
header: Connection: close
header: Transfer-Encoding: chunked
[youtube] DRRttXwx5yY: Downloading video info webpage
send: u'GET /get_video_info?ps=default&el=info&video_id=DRRttXwx5yY&hl=en&sts=17913&disable_polymer=true&gl=US&eurl= HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: YSC=Z-mDXy9OhLY; VISITOR_INFO1_LIVE=YE9rggcKOH4; GPS=1; PREF=f1=50000000&hl=en; s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/x-www-form-urlencoded
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
header: Content-Length: 50
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Cache-Control: no-store
header: Strict-Transport-Security: max-age=31536000
header: Date: Sun, 20 Jan 2019 22:45:09 GMT
header: Server: YouTube Frontend Proxy
header: Set-Cookie: s_gl=1d69aac621b2f9c0a25dade722d6e24bcwIAAABVUw==; path=/; domain=.youtube.com
header: Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
header: Connection: close

^^^ and stuck here

Description of your issue, suggested solution and other information

youtube-dl seems to stall / slow down, especially on larger videos. I have been able to reproduce the issue on two different connections (webpass and a cogent). The Downloading Webpage and Downloading video info webpage take anywhere from a relatively long time to infinity, sometimes hanging there. I thought it was related to #15914 though . Adding rate limiting via -r 100K doesn't seem to do anything. I can not determine if this is a connection issue with ISPs throttling / shaping traffic or something youtube is doing.

Relevant or not relevant things:
No Proxies or any weird internet settings
DNS is configured to 1.1.1.1 and 8.8.8.8
OSX 10.12.6 (16G1510)
Clean hosts file (though I've experimented with adding youtube.com and googlevideo.com dns entries)
Relatively fast pipe/internet connection:

$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Webpass (136.24.56.69)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Unwired (San Francisco, CA) [3.02 km]: 5.908 ms
Testing download speed................................................................................
Download: 359.91 Mbit/s
Testing upload speed................................................................................................
Upload: 454.84 Mbit/s

Python versions

$ python --version
Python 2.7.15
$ python3 --version
Python 3.7.2

DNS cache log while running sudo killall -INFO mDNSResponder

$ log stream --predicate 'process == "mDNSResponder"' --info | grep -E 'youtube|googlevideo'
2019-01-20 14:51:53.158495-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  13      1843 -U-      CNAME   36 r2---sn-fxc25nn-nwje.googlevideo.com. CNAME r2.sn-fxc25nn-nwje.googlevideo.com.
2019-01-20 14:51:53.158532-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 216.58.194.174
2019-01-20 14:51:53.158566-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 216.58.194.206
2019-01-20 14:51:53.158620-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 216.58.195.78
2019-01-20 14:51:53.158659-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 216.58.195.238
2019-01-20 14:51:53.158695-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 172.217.0.46
2019-01-20 14:51:53.158726-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 172.217.5.110
2019-01-20 14:51:53.158781-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 172.217.6.78
2019-01-20 14:51:53.158822-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 172.217.164.110
2019-01-20 14:51:53.158872-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       168 -U-      Addr     4 youtube-ui.l.google.com. Addr 216.58.192.14
2019-01-20 14:51:53.158911-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO]  81       221 -U-      AAAA    16 youtube-ui.l.google.com. AAAA 2607:F8B0:4005:0807:0000:0000:0000:200E
2019-01-20 14:51:53.158938-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 110       651 -U-      Addr     4 r1.sn-fxc25nn-nwje.googlevideo.com. Addr 208.87.60.76
2019-01-20 14:51:53.158968-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 110      1637 -U-      AAAA    16 r1.sn-fxc25nn-nwje.googlevideo.com. AAAA 2604:5500:0003:000D:0000:0000:0000:000C
2019-01-20 14:51:53.159172-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 216      6267 -U-      CNAME   25 www.youtube.com. CNAME youtube-ui.l.google.com.
2019-01-20 14:51:53.159451-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 311      1221 -U-      CNAME   36 r1---sn-fxc25nn-nwje.googlevideo.com. CNAME r1.sn-fxc25nn-nwje.googlevideo.com.
2019-01-20 14:51:53.159959-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 447      1843 -U-      AAAA    16 r2.sn-fxc25nn-nwje.googlevideo.com. AAAA 2604:5500:0003:000D:0000:0000:0000:000D
2019-01-20 14:51:53.159994-0800 0x4ab      Default     0x0                  168    mDNSResponder: [com.apple.mDNSResponder.AllINFO] 447      1783 -U-      Addr     4 r2.sn-fxc25nn-nwje.googlevideo.com. Addr 208.87.60.77
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Jan 21, 2019

Nothing to do with DNS flood as it's not a segmented download.
Run it under strace to figure out the reason of a stall.
@dstftw dstftw closed this Jan 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cant-reproduce
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
@diziet @dstftw
You can’t perform that action at this time.