HTTP Error 429: Too Many Requests (caused by HTTPError()); [youtube]

[bussiere]

My version is the last one 💯
youtube-dl --version
2015.03.03.1

youtube-dl -f 22 https://www.youtube.com/watch?v=8MMnbeGDzdw
[youtube] 8MMnbeGDzdw: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Thanks for the program and regards

[marburger93]

I have the same problem with youtube. HTTP requests without youtube-dl are possible

~# youtube-dl https://www.youtube.com/watch?v=FS2EZ18Fcv8
[youtube] FS2EZ18Fcv8: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
~# youtube-dl -U
youtube-dl is up-to-date (2015.03.03.1)

[TheBoroer]

Same issue here. Requests without youtube-dl are working fine.

URL used: http://www.youtube.com/watch?v=PKvg5PNPihY

Verbose Output:

[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', '-s', '-g', 'http://www.youtube.com/watch?v=PKvg5PNPihY']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.03.03.1
[debug] Python version 2.6.6 - Linux-3.10.23-xxxx-std-ipv6-64-x86_64-with-centos-6.5-Final
[debug] exe versions: ffmpeg 0.6.5, ffprobe 0.6.5
[debug] Proxy map: {}
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "./youtube-link/youtube_dl/extractor/common.py", line 314, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "./youtube-link/youtube_dl/YoutubeDL.py", line 1674, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[sargue]

+1

My output:

$ youtube-dl https://www.youtube.com/watch?v=j3mhkYbznBk --verbose
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['https://www.youtube.com/watch?v=j3mhkYbznBk', '--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.03.03.1
[debug] Python version 2.7.3 - Linux-3.2.13-xxxx-std-ipv6-32-i686-with-debian-7.8
[debug] exe versions: avconv 0.8.16-6, avprobe 0.8.16-6, ffmpeg 0.8.16-6, ffprobe 0.8.16-6
[debug] Proxy map: {}
[youtube] j3mhkYbznBk: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/home/sargue/bin/youtube-dl/youtube_dl/extractor/common.py", line 314, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/home/sargue/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1674, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python2.7/urllib2.py", line 407, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 445, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

[phihag]

Everything is working fine for me from multiple machines all over the world, so I cannot diagnose this issue. Can anyone of you offer a proxy or VPN at an affected machine? If so, please contact me.

[Mazwak]

It seems to be a server side throttling with a recaptcha.

I have no idea how it can be bypassed.

[bo3rnd]

It can only be bypassed with an other IP.

BTW: I have a server at OVH and it seems their whole IPv6 address space is blacklisted, so changing to an other IPv6 address does not work. Only IPv4 does work.

[Mazwak]

I do proxy through OVH also.
Loading exported firefox cookies into youtube-dl, after entering the captcha in FF, works.

I tried without ovh proxying, and it is the same.

[Aiexis]

Did someone found a solution for OVH ? Tried 2 VPS, and same issue.

[TheBoroer]

I opened a ticket with OVH support and they escalated it. I think it might be an issue with youtube blocking all OVH ipv6 blocks (since ipv4 works fine).

[Aiexis]

Is there a way to user ipv4 instead of ipv6 ? (configuring debian maybe ?)

[TheBoroer]

@Aiexis youtube-dl has -4 and -6 parameters that you can use to tell youtube-dl to run it through ipv4 or ipv6. Check the readme/docs

[Aiexis]

Thank you ;)
-4 parameter works on one VPS (but not the second)

[jefi009]

@blackpropeller Did you get any reply from OVH so far? I've opened a ticket as well and nobody took care of it so far.

[TheBoroer]

@jefi009 Yup, they replied but they weren't really that helpful.

After investigating the issue with our network specialists and testing the command from multiple sources inside and outside of our data centres, we can see that the issue is on youtube's side. From what we understand, they are trying to prioritize the use of their API instead. For more information on the issue, I would suggest contacting them directly.

They pretty much said we're on our own...lol.

[TheBoroer]

@isez I had a feeling OVH couldnt help but i did find it odd that OVH's entire IPv6 block was blocked by youtube (even on servers/systems that have never made a single request to youtube). They had this sort of issue before with IPv4 and were able to sort the issue out with youtube. That's why i asked them to look into it again but for IPv6 this time.

The cookies option/solution works fine for now i guess.

[jefi009]

@blackpropeller It sounds like they didn't even contact YT. I also noticed that their whole Range is blocked since I've also tried to use servers that never made a request to YT.

My guts tell me that it is more likely that someone started a DDOS from within the OVH network towards yt which triggered an automatic system at YT that blocked the OVH range. Their explanation would only make sense if their IPv4 ranges would be blocked as well.

[jefi009]

I did also notice that the 429's appeared one week before and lasted only a few hours, on March 5th the 429's also disappeared temporarily after a few hours only to reappear permanently for now.

This could indicate

A) A test carried out by YT.
B) An automatic DOS protection that increases the block time after each block.

[bo3rnd]

Not only OVH is affected, it seems the whole IPv6 space from online.net (another popular french provider) is also blacklisted.

[Utopiah]

For those using YoutubeDL.py setting source_address with the IPv4 in use works (no IPv4 parameter seen in ydl_opts{}).

[renaudallard]

I can confirm what bo3rnd says. My IPv6 range at online.net is also blocked by this and I get a captcha. But as jefi009 said, this makes no sense as they should also blacklist ipv4 ranges if it was to block servers. In my case, I use my ipv6 range at online to get an ipv6 for my home machines, so this is really a hassle.

[pitrou]

Same here on a OVH server, and if I force IPv4 (using the -4 option) it works.

[sisihagen]

Yes with ipv4 goes well with ovh servers....

[celogeek]

The IPV4 lead to a video that explain you need to do something with the captcha. But still the same issue.

[chovy]

using -4 option on OVH works

[yarwelp]

Using a VPS provided by Vultr, requests are blocked on both IPv4 and IPv6. I think we should develop a ReCAPTCHA proxy tool to make it fast and easy to solve a captcha and use the cookie. That might be outside the scope of youtube-dl, though.

[itslukej]

"-4" and "-6" still seems to be very experimental... (it's not forcing it)

[danwdart]

No good from my DigitalOcean droplet either :/
Going to try proxying through the machine and entering the captcha myself...
Although this is never going to be any good if this keeps happening.

[rudolphos]

@bo3rnd why did they blacklist OVH servers ? Crooked Google as always trying to prevent people from downloading videos..

[danwdart]

I think it's against the terms of service not to use their own viewer tbh. Bad but they don't want you to watch the content without their ads.

Which is why they only allow CC-BY (in order to relicence it so that you're not allowed to download it).

[yarwelp]

@dandart CC-BY is the most liberal CC license short of CC0. This license does not imply that you're not allowed to download it, quite the opposite. CC-BY allows others to take a work someone has made and to redistribute it with or without modification as long as you attribute the original author.

Meanwhile, YouTube may have some conditions about downloading in their ToS. I dunno, I haven't read the ToS. Usually conditions about downloading are about large-scale, automated downloading of resources (such as videos) and/or scraping of content like titles, descriptions and comments.

If anyone knows the official stand YouTube has, if any, I'd like to hear it.

[danwdart]

Well, CC-BY is the most liberal in that it is not copyleft: i.e. you can still relicence it to say that you're not allowed to download it! This is what I mean when I think YouTube is doing that.

If other CC-licenced videos were allowed on YouTube (such as any -SA) then YouTube wouldn't be able to stop people downloading, and as it is, they don't support anything that stops them blocking you from downloading or stops them making money off of it, obviously for ads.

Their ToS says that one is not allowed to download from them, just at all.

I've tweeted them but not yet got a response.

[danwdart]

I wonder - has anyone tried it with authentication? Maybe it'll work with an API key or a un/pw login cookie?

[i2mo]

same here on a DigitalOcean Host located in Frankfurt. IPv6 is blocked, v4 is working.
youtube-dl -4 is working then.

[tjg1]

It's unfortunate, but I can understand why youtube does this. Since providers give their customers large IPv6 block allocations, it's not feasible to block abusers by their IPv6 address - they can simply change or randomise it.

Oh, and the bug here is that youtube-dl tries using IPv6 even when youtube is unreachable over IPv6. (Enabling IPv6 over OpenVPN requires additional configuration).