Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Youtube] HTTP Error 404: Not Found (caused by HTTPError()) on Python 2 #8918

Open
ewnd9 opened this issue Mar 21, 2016 · 18 comments
Open

[Youtube] HTTP Error 404: Not Found (caused by HTTPError()) on Python 2 #8918

ewnd9 opened this issue Mar 21, 2016 · 18 comments

Comments

@ewnd9
Copy link

@ewnd9 ewnd9 commented Mar 21, 2016

$ youtube-dl "https://www.youtube.com/watch?v=UdO7_GrRttM" --verbose

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=UdO7_GrRttM', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.03.18
[debug] Python version 2.7.6 - Linux-3.16.0-60-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] UdO7_GrRttM: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 365, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1929, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Anything else I should attach?

@rg3
Copy link
Collaborator

@rg3 rg3 commented Mar 21, 2016

It works for me here. Can you check it was not a transient error on YouTube or your ISP?

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

Could you elaborate, please (e.g. command to run in terminal to check this)? If you mean is there any errors regardless internet connection itself, everything is working okay except youtube-dl

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

Headers from accessing url by curl

$ curl -s -D - "https://www.youtube.com/watch?v=UdO7_GrRttM" -o /dev/null
HTTP/1.1 200 OK
X-Frame-Options: SAMEORIGIN
Expires: Tue, 27 Apr 1971 19:44:06 EST
X-Content-Type-Options: nosniff
P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=ru for more info."
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
X-XSS-Protection: 1; mode=block; report=https://www.google.com/appserve/security-bugs/log/youtube
Date: Mon, 21 Mar 2016 13:42:11 GMT
Server: Ytfe_Worker
Set-Cookie: YSC=KLHg8PeOytk; path=/; domain=.youtube.com; httponly
Set-Cookie: PREF=f1=50000000; path=/; domain=.youtube.com; expires=Sun, 20-Nov-2016 01:35:11 GMT
Set-Cookie: VISITOR_INFO1_LIVE=Ofpd47IQUwM; path=/; domain=.youtube.com; expires=Sun, 20-Nov-2016 01:35:11 GMT; httponly
Alternate-Protocol: 443:quic,p=1
Alt-Svc: quic=":443"; ma=2592000; v="31,30,29,28,27,26,25"
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked

I've remembered that around a month ago youtu.be domain was blocked by Russian government but it is available now. Are there any non-standard domains involved which I could check? (not youtube.com and youtu.be

@rg3
Copy link
Collaborator

@rg3 rg3 commented Mar 21, 2016

What I meant is that the video downloads fine here. You should probably wait for another team member to help you but, in the mean time, can you post here the program output if you add --dump-pages to the command you run? Thanks.

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

$ youtube-dl "http://www.youtube.com/watch?v=UdO7_GrRttM" --verbose --dump-pages 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'http://www.youtube.com/watch?v=UdO7_GrRttM', u'--verbose', u'--dump-pages']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.03.18
[debug] Python version 2.7.6 - Linux-3.16.0-60-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] UdO7_GrRttM: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 365, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1929, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

I've changed url from https to http to exclude assumption of ssl problems, commands from my previous messages show same results with http url.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 21, 2016

Post the output of youtube-dl --print-traffic http://www.youtube.com/watch?v=UdO7_GrRttM.

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

$ youtube-dl --print-traffic http://www.youtube.com/watch?v=UdO7_GrRttM  
[youtube] UdO7_GrRttM: Downloading webpage
send: u'GET /watch?v=UdO7_GrRttM&gl=US&hl=en&has_verified=1&bpctr=9999999999 HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nHost: www.youtube.com\r\nCookie: PREF=f1=50000000&hl=en\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Mon, 21 Mar 2016 16:37:36 GMT
header: Server: Apache/2.2.22
header: Last-Modified: Mon, 01 Feb 2016 08:25:06 GMT
header: ETag: "4080d-72a-52ab11f519717;52ab116f5e177"
header: Accept-Ranges: bytes
header: Vary: Accept-Encoding
header: Content-Encoding: gzip
header: Content-Length: 1100
header: Connection: close
header: Content-Type: text/html
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 21, 2016

What's the output of
curl 'http://www.youtube.com/watch?v=UdO7_GrRttM&gl=US&hl=en&has_verified=1&bpctr=9999999999'
and
curl -A 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)' 'http://www.youtube.com/watch?v=UdO7_GrRttM&gl=US&hl=en&has_verified=1&bpctr=9999999999'?

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Mar 21, 2016

Can you watch it in browser at all?

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

curl 'http://www.youtube.com/watch?v=UdO7_GrRttM&gl=US&hl=en&has_verified=1&bpctr=9999999999'

http://pastebin.com/ui077RJx

curl -A 'Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)' 'http://www.youtube.com/watch?v=UdO7_GrRttM&gl=US&hl=en&has_verified=1&bpctr=9999999999'

http://pastebin.com/tgYFjKs1

Can you watch it in browser at all?

Yes

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

Made some code experiments

import urllib2
response = urllib2.urlopen('https://www.youtube.com/watch?v=UdO7_GrRttM')
html = response.read()
print(html)
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    response = urllib2.urlopen('https://www.youtube.com/watch?v=UdO7_GrRttM')
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
import urllib2
response = urllib2.urlopen('https://www.youtube.com/')
html = response.read()
print(html)
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Ссылка заблокирована</title>
<style type="text/css">
<!--
H5 {font-size:15px; font-weight:normal; letter-spacing:1px; text-transform:uppercase; white-space:nowrap; margin-bottom:14px; font-weight:bold;}
BODY {color:#4d4d9f; font-size:13px; background: #FFFFFF;  width: 99%;}
DIV.container {width:50%;margin-top:20px; margin:100px auto; background:#FFFFFF; border:3px solid #E42319; padding:10px; -moz-border-radius:20px; -khtml-border-radius: 20px; -webkit-border-radius: 20px; border-radius:20px;}
-->
</style>
</head>
<body>
<div class="container">
<center><img onerror="this.style.display='none';" src="http://img.ytapi.com/ttk.png">
<h5>Ссылка заблокирована!!!<br>в соответствии с законодательством РФ<br>Причина блокировки: <a href="http://blocklist.rkn.gov.ru/">blocklist.rkn.gov.ru</a></h5><br><br><a href="http://ttk.ru/">ЗАО "Компания ТрансТелеКом"</a>
</center></div>
<script type="text/javascript"> (function (d, w, c) { (w[c] = w[c] || []).push(function() { try { w.yaCounter33269963 = new Ya.Metrika({ id:33269963, clickmap:true, trackLinks:true, accurateTrackBounce:true }); } catch(e) { } }); var n = d.getElementsByTagName("script")[0], s = d.createElement("script"), f = function () { n.parentNode.insertBefore(s, n); }; s.type = "text/javascript"; s.async = true; s.src = "https://mc.yandex.ru/metrika/watch.js"; if (w.opera == "[object Opera]") { d.addEventListener("DOMContentLoaded", f, false); } else { f(); } })(document, window, "yandex_metrika_callbacks");</script><noscript><div><img src="https://mc.yandex.ru/watch/33269963" style="position:absolute; left:-9999px;" alt="" /></div></noscript>
</body></html>

It says that url is blocked by the government (so now 404 makes sense), but somehow google-chrome, curl and node.js are displaying youtube itself (http://youtube.com/).

Any ideas how could python's urllib2 use cached dns?

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Mar 21, 2016

python 3 is working ok

import urllib.request

url = 'https://www.youtube.com/'
request = urllib.request.urlopen(url)

print(request.read())

and, when I set python3 as the default python in the system ($ sudo mv /usr/bin/python3 /usr/bin/python), youtube-dl started working too

$ youtube-dl "https://www.youtube.com/watch?v=UdO7_GrRttM" --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=UdO7_GrRttM', '--verbose']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.03.18
[debug] Python version 3.4.3 - Linux-3.16.0-60-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.18-6, avprobe 9.18-6, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] UdO7_GrRttM: Downloading webpage
[youtube] UdO7_GrRttM: Downloading video info webpage
[youtube] UdO7_GrRttM: Extracting video information
[youtube] UdO7_GrRttM: Downloading MPD manifest
WARNING: Your copy of avconv is outdated and unable to properly mux separate video and audio files, youtube-dl will download single file media. Update avconv to version 10-0 or newer to fix this.
[debug] Invoking downloader on 'https://r7---sn-ug5onuxaxjvh-n8vz.googlevideo.com/videoplayback?mime=video%2Fmp4&key=yt6&itag=22&ipbits=0&sver=3&lmt=1458046689112654&signature=13AA167332E9E50715862938C14AFFCC16C267CB.60E23084B322E25A920E4748ABE4B678C5DAEE2C&mm=31&source=youtube&mn=sn-ug5onuxaxjvh-n8vz&ratebypass=yes&dur=69.752&mt=1458588690&mv=m&ms=au&fexp=9405984%2C9407191%2C9413142%2C9416126%2C9417828%2C9419817%2C9420452%2C9422596%2C9423661%2C9423662%2C9427902%2C9428422%2C9429160%2C9429808%2C9431012%2C9431270%2C9431400%2C9431619%2C9432057%2C9432437&ip=62.33.207.199&id=o-AIYuUW13dt4a3nKRAj_XOpEiexMAKv-zDwgl7trvxT2Y&sparams=dur%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Cratebypass%2Crequiressl%2Csource%2Cupn%2Cexpire&initcwndbps=3792500&expire=1458610418&upn=dmr9Dx8UCm0&pl=21&requiressl=yes'
[download] Destination: Unity Launcher And Dash Can Now Be Moved To The Bottom Of The Screen [Ubuntu 16.04 Xenial Xerus]-UdO7_GrRttM.mp4
[download] 100% of 4.85MiB in 00:00
@abhigenie92
Copy link

@abhigenie92 abhigenie92 commented Oct 19, 2016

Similar issue.

youtube-dl --verbose "https://www.youtube.com/watch\?v\=onBYsen2_eA"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'https://www.youtube.com/watch\\?v\\=onBYsen2_eA']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.06.25
[debug] Python version 2.7.12+ - Linux-4.8.0-22-generic-x86_64-with-Ubuntu-16.10-yakkety
[debug] exe versions: ffmpeg 3.0.2-1ubuntu3, ffprobe 3.0.2-1ubuntu3, rtmpdump 2.4
[debug] Proxy map: {}
[generic] watch\?v\=onBYsen2_eA: Requesting header
WARNING: Could not send HEAD request to https://www.youtube.com/watch\?v\=onBYsen2_eA: HTTP Error 404: Not Found
[generic] watch\?v\=onBYsen2_eA: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 390, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1950, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

 ✘  ~/Desktop/Lectures/CS/AI/monte_claro  apt list | grep youtube

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libwebservice-youtube-perl/yakkety,yakkety 1.0.3-3 all
libwww-youtube-download-perl/yakkety,yakkety 0.59-1 all
mopidy-youtube/yakkety,yakkety 2.0.0-2 all
mps-youtube/yakkety,yakkety 0.2.7.1-1 all
nuvolaplayer3-youtube/now 1.2-1~xenial amd64 [installed,local]
python-sphinxcontrib.youtube/yakkety,yakkety 1.0-1 all
python3-sphinxcontrib.youtube/yakkety,yakkety 1.0-1 all
unity-webapps-youtube/yakkety,yakkety 2.4.16+16.04.20151119-0ubuntu1 all
youtube-dl/yakkety,yakkety,now 2016.06.25-2 all [installed]
 ~/Desktop/Lectures/CS/AI/monte_claro  

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Nov 12, 2016

@abhigenie92 There's no need to escape special characters in quotation marks. The following command should work fine:

youtube-dl --verbose "https://www.youtube.com/watch?v=onBYsen2_eA"
@rkfg
Copy link

@rkfg rkfg commented Jan 5, 2017

@ewnd9 thank you for the python3 trick. It worked for me for some reason. That's a Russian censorship related issue but I don't understand the mechanism of urllib2 failing. I can browse YouTube perfectly fine with browser but at some point youtube-dl started failing. In Ubuntu/Debian it's sufficient to uninstall youtube-dl and then install pip3 with apt-get and do pip3 install youtube-dl

@yan12125 yan12125 changed the title [Youtube] HTTP Error 404: Not Found (caused by HTTPError()) [Youtube] HTTP Error 404: Not Found (caused by HTTPError()) on Python 2 Jan 6, 2017
@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Jan 6, 2017

@rkfg As I see it now, urlib2 is either caching by itself or using a system DNS cache mechanism. Cache was set when youtube was blocked, so it still loads block-page from ISP.

I believe you can simply run youtube-dl binary with python3 like $ python3 /usr/bin/youtube-dl instead of reinstalling

@rkfg
Copy link

@rkfg rkfg commented Jan 6, 2017

I believe you can simply run youtube-dl binary with python3 like $ python3 /usr/bin/youtube-dl instead of reinstalling

Nope, tried it right after I found this issue here. The libraries are installed to /usr/local/lib/python2.7/dist-packages/ so they're not found by python3. Reinstall did the trick just fine.

@ewnd9
Copy link
Author

@ewnd9 ewnd9 commented Jan 6, 2017

Thanks for the clarification 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.