ERROR: Unable to download webpage

[Nihim]

Worked fine till yesterday when I upgraded to the latest. I keep getting:

bin/youtube-dl --verbose -F https://www.youtube.com/watch?v=UdAapVZFS1g
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'-F', u'https://www.youtube.com/watch?v=UdAapVZFS1g']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.06.25
[debug] Python version 2.7.3 - Linux-3.16.0-0.bpo.4-amd64-x86_64-with-debian-7.8
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] UdAapVZFS1g: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/nihim/.psources/youtube-dl/local/lib/python2.7/site-packages/youtube_dl/extractor/common.py", line 312, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/nihim/.psources/youtube-dl/local/lib/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 1729, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 407, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 445, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Kindly keep in mind that I am a ~new at nix so it might be something obvious that I am missing.

Cheers

[remitamine]

the output show that the error happen because the server return an HTTP error code.

HTTP Error 429: Too Many Requests.

as the RFC 6585:

The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting").

maybe you are rquesting many videos from youtube in a short time.

[Nihim]

It gave me that error from the first time and I hadn't used it for days or even a week+.
Also I tried to re-update to latest and retry to grab a link another 2-3 times in a 15h period.

[jaimeMF]

Are you using youtube-dl using some web hosting or using some proxy/VPN?

[Nihim]

Nope dedicated box, no proxy / VPN. As I said it always worked fine and since I rarely use it I always upgrade to latest before dling something (since that's what it recommends itself).

[jaimeMF]

Can you watch the video on a browser using the same computer?
Maybe there's other person in the same network downloading a lot of videos?

[Nihim]

it's cli OS and it's on one of the biggest DCs in europe. Ofcourse it has it's own dedicated IPv4 / IPv6 range.

[jaimeMF]

Well, you could try running wget 'https://www.youtube.com/watch?v=UdAapVZFS1g' (or using curl) and see what happens, it could be usefult to inspect the webpage they get.
I asked about the network and other stuff because a user had the same problem when running youtube-dl in AWS (#6072) and others can't use it with OVH (#5138).

[Nihim]

HTTP request sent, awaiting response... 429 Too Many Requests

the plot thickens. So maybe yea youtube has blacklisted the DCs IP ranges, which makes it throw that captcha.

Gonna make a ssh tunnel on that box and try to visit the page see if I get the captcha.

[Nihim]

Nope works fine. But still prolly is youtube blacklisting the IPs?

[jaimeMF]

Nope works fine

What works fine?
If wget/curl can't download the webpage, youtube-dl can't do it.

[Nihim]

Agreed but it does work through ssh tunnel (and ofc I checked what IP was reported to make sure I was indeed using the ssh tunnel). It just went right into the page and started loading the video.

Doesn't wget and other dl tools report something like a type of client? Maybe they are blocking that. I remember one site was blocking python / perl based get scripts. They were reporting some kind of user / client / whatever type which they were effectively blocking.

Sorry what I am saying is so vague but I don't remember the exact details and couldn't find the logs of the conversation I had with an admin of that site that explained to me why and what happened.

[jaimeMF]

It just went right into the page and started loading the video.

Could you play it directly or did you had to solve some captcha?

Doesn't wget and other dl tools report something like a type of client? Maybe they are blocking that. I remember one site was blocking python / perl based get scripts. They were reporting some kind of user / client / whatever type which they were effectively blocking.

They could look into the user agent header, but the youtube-dl UA looks like a normal browser and it doesn't contain any reference to youtube-dl (run youtube-dl --dump-user-agent). You could try running curl -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:39.0) Gecko/20100101 Firefox/39.0' YOUTUBE_URL (you can get the user agent of your browser by running navigator.userAgent in the developer tools) or using the --user-agent option in youtube-dl to see if that works.

[Nihim]

No captcha straight to the video (used private browsing incase my cookies bypassed it).

curl -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:39.0) Gecko/20100101 Firefox/39.0' youtube link
throwed out the source page code.

ps: what's the code for quoting? :P

youtube-dl --dump-user-agent:
Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)

bin/youtube-dl --user-agent 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:39.0) Gecko/20100101 Firefox/39.0' 'link'
failed with the same error

[jaimeMF]

For wget you have to use -U, but I guess you would get the same error.
The other possibility is they use javascript to load the page. Just to confirm, are you 100% sure that when using the browser you are actually using the network from the other machine? (I don't know how you setup it, so I don't know how easy is it to do it correctly).

[Nihim]

wget same error.

What I did was just google whatsmyip and verified that the IP shown was the servers and not of my home.
As for the method it's a simple ssh tunnel & firefox running on proxy mode so it routes it's traffic through the ssh tunnel.

Maybe I am doing something wrong but :/

[ishupreet]

I am also getting the same, i have installed an Ubuntu instance on AWS, and when i wget i get:
HTTP request sent, awaiting response... 429 Too Many Requests
2015-07-03 17:50:33 ERROR 429: Too Many Requests.

[jaimeMF]

@ishupreet see #6072.