Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) #9802

dorexpro · 2016-06-16T09:52:20Z

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'--print-traffic', u'--print-json', u'-o', u'/home/media/youtube/%(title)s.%(ext)s', u'http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.14
[debug] Python version 2.7.6 - Linux-3.13.0-88-generic-i686-with-Ubuntu-14.04-trusty
[debug] exe versions: ffmpeg 2.5.git, ffprobe 2.5.git
[debug] Proxy map: {}
send: u'GET /playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,_;q=0.7\r\nHost: www.youtube.com\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Thu, 16 Jun 2016 09:44:40 GMT
header: Server: Apache/2.2.22
header: Last-Modified: Mon, 01 Feb 2016 08:25:06 GMT
header: ETag: "4080d-72a-52ab11f519717;534ad8455a74d"
header: Accept-Ranges: bytes
header: Vary: Accept-Encoding
header: Content-Encoding: gzip
header: Content-Length: 1100
header: Connection: close
header: Content-Type: text/html
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/lib/python2.7/dist-packages/youtube_dl-2016.06.14-py2.7.egg/youtube_dl/extractor/common.py", line 390, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/lib/python2.7/dist-packages/youtube_dl-2016.06.14-py2.7.egg/youtube_dl/YoutubeDL.py", line 1950, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 416, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 529, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 454, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 383, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 537, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

yan12125 · 2016-06-16T12:03:39Z

Can you play this playlist in browsers?

dorexpro · 2016-06-16T12:09:06Z

Yes.
Sorry, I just read in English. I translate it with Google.
The browser opens a list of normal. Using Python 3.4, is also normal. Error when using Python 2.7.

yan12125 · 2016-06-16T12:35:34Z

Here's my result:

$ youtube-dl python2 -v "http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg" --print-traffic
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg', u'--print-traffic']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.14
[debug] Git HEAD: efbd1eb
[debug] Python version 2.7.11 - Linux-4.6.2-1-ARCH-x86_64-with-glibc2.2.5
[debug] exe versions: avconv v12_dev0-2785-g1e9c5bf, avprobe v12_dev0-2785-g1e9c5bf, ffmpeg 3.0.2, ffprobe 3.0.2, rtmpdump 2.4
[debug] Proxy map: {}
[youtube:playlist] FLpFyQ_IxKBvBzKyP1uijeEg: Downloading webpage
send: u'GET /playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: X-Frame-Options: SAMEORIGIN
header: Strict-Transport-Security: max-age=604800
header: Cache-Control: no-cache
header: Content-Type: text/html; charset=utf-8
header: Content-Encoding: gzip
header: X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
header: Expires: Tue, 27 Apr 1971 19:44:06 EST
header: X-Content-Type-Options: nosniff
header: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=en for more info."
header: Date: Thu, 16 Jun 2016 12:29:13 GMT
header: Server: YouTubeFrontEnd
header: Set-Cookie: VISITOR_INFO1_LIVE=TN2rFg8eNLU; path=/; domain=.youtube.com; expires=Wed, 15-Feb-2017 00:22:12 GMT; httponly
header: Set-Cookie: PREF=f1=50000000; path=/; domain=.youtube.com; expires=Wed, 15-Feb-2017 00:22:13 GMT
header: Set-Cookie: YSC=FtCXliIQr3c; path=/; domain=.youtube.com; httponly
header: Connection: close
header: Transfer-Encoding: chunked
[download] Downloading playlist: Favorites
[youtube:playlist] playlist Favorites: Downloading 0 videos
[download] Finished downloading playlist: Favorites

See this line:

header: Server: YouTubeFrontEnd

And a similar line in your log:

header: Server: Apache/2.2.22

Apache 2.2.22 is quite old and I don't thing YouTube is using it. I guess the connecton between your device and YouTube servers is intercepted. youtube-dl's requests are not really sent to YouTube servers.

siddht4 · 2016-06-16T22:02:33Z

use youtube-dl directly , dont use the link generated through google translate

eg: youtube-dl https://translate.google.co.in/translate?hl=en&sl=ru&tl=en&u=https%3A%2F%2Fwww.youtube.com%2Fuser%2Fdorexspb fails

but youtube-dl www.youtube.com%2Fuser%2Fdorexspb works fine

i also see that the youtube-dl you are using is a developer build one , delete the youtube-dl from /usr/local/bin/youtube-dl location and use this

sudo wget https://yt-dl.org/downloads/2016.06.16/youtube-dl -O /usr/local/bin/youtube-dl

sudo chmod a+rx /usr/local/bin/youtube-dl

and then try again youtube-dl sometimes fails to work if you just installed from a secondary source

eg : ppa + dev

dorexpro · 2016-06-17T05:12:15Z

I use a translator only for this forum. Do not call for youtube-dl.
developer build of the use of this error. This is an attempt to solve the problem.

yan12125 · 2016-06-17T09:17:01Z

Well, could you paste the Python 3 log, too?

dorexpro · 2016-06-17T09:25:30Z

root@serv:/home/youtube# youtube-dl -v --print-traffic -o "/home/media/youtube/%(title)s.%(ext)s" "http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--print-traffic', '-o', '/home/media/youtube/%(title)s.%(ext)s', 'http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.14
[debug] Python version 3.4.3 - Linux-3.13.0-88-generic-i686-with-Ubuntu-14.04-trusty
[debug] exe versions: ffmpeg 2.5.git, ffprobe 2.5.git
[debug] Proxy map: {}
[youtube:playlist] FLpFyQ_IxKBvBzKyP1uijeEg: Downloading webpage
[download] Downloading playlist: Favorites
[youtube:playlist] playlist Favorites: Downloading 0 videos
[download] Finished downloading playlist: Favorites

dorexpro · 2016-06-17T09:27:11Z

This error on the forum has already been described. The solution was to move to Python 3.

yan12125 · 2016-06-17T09:30:16Z

Can't find any difference...Could you run the following commands:

$ strace -f python2 youtube-dl "http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg" >& python2_log.txt
$ strace -f python3 youtube-dl "http://www.youtube.com/playlist?list=FLpFyQ_IxKBvBzKyP1uijeEg" >& python3_log.txt

And upload python2_log.txt and python3_log.txt?

dorexpro · 2016-06-17T09:38:06Z

python3_log.txt
python2_log.txt

yan12125 · 2016-06-17T09:42:05Z

Both said:

can't open file 'youtube-dl': [Errno 2] No such file or directory

Could you replace youtube-dl in my previous command with the full path of youtube-dl?

dorexpro · 2016-06-17T10:13:34Z

python3_log.txt
python2_log.txt

yan12125 · 2016-06-17T10:47:24Z

The problem is quite comlex:

Your DNS server (192.168.1.1) returns two IPs for www.youtube.com - 188.43.20.69 and 62.33.207.199. These IPs look like official Google servers.
Both servers require a client feature called "Server Name Indication" (SNI). Clients without SNI always get 404 errors.

I guess the only solution is using Python versions with SNI. If you need 2.x, Python 2.7.9 or newer has SNI.

dorexpro · 2016-06-17T10:54:21Z

Thank you very much!

dntt1 · 2016-06-17T11:58:51Z

well i see both the external ips are blocked in russia , got this error
Ссылка заблокирована!!!
в соответствии с законодательством РФ
Причина блокировки: blocklist.rkn.gov.ru

use a proxy to download from youtube

i dont see this ip as of any domain youtube or google .
maybe your isp is redirecting as a portal gateway.
can`t really say as its blocked

yan12125 added the cant-reproduce label Jun 16, 2016

yan12125 removed the cant-reproduce label Jun 17, 2016

yan12125 closed this Jun 17, 2016

yan12125 added the external-bugs label Jun 17, 2016

yan12125 changed the title ~~Unable to download webpage: HTTP Error 404: Not Found~~ Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) Jun 17, 2016

ytdl-org / youtube-dl

Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) #9802

Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) #9802

dorexpro commented Jun 16, 2016

yan12125 commented Jun 16, 2016

dorexpro commented Jun 16, 2016

yan12125 commented Jun 16, 2016

siddht4 commented Jun 16, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

dntt1 commented Jun 17, 2016

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) #9802

Some YouTube servers return HTTP 404 to clients without SNI (was: Unable to download webpage: HTTP Error 404: Not Found) #9802

Comments

dorexpro commented Jun 16, 2016

yan12125 commented Jun 16, 2016

dorexpro commented Jun 16, 2016

yan12125 commented Jun 16, 2016

siddht4 commented Jun 16, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

yan12125 commented Jun 17, 2016

dorexpro commented Jun 17, 2016

dntt1 commented Jun 17, 2016

Essential cookies

Always active

Analytics cookies