Youtube Error 429 Too many requests. Ways to slow down crawling?

[bathrobehero]

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Hi, recently I keep running into Error 429: Too many request on Youtube even with 2 minutes between checking a list of channels for new videos and even with --sleep-interval set. And EVEN if I change the IP, it comes back soon.

It's usually followed after "Downloading video info webpage".

So I wonder, can I somehow slow down the speed at which youtube-dl crawls channels/videos?
I don't want to rate limit downloads, that's not the issue, the issue seems to be crawling too fast.

[oiueaieouaeuoauiaoei]

Probably related, but (for me) the four /get_video_info requests always respond with a 429, even after throttling every network requests to one per minute.

youtube-dl/youtube_dl/extractor/youtube.py

Lines 1776 to 1792 in 42cd082

	for el in ('embedded', 'detailpage', 'vevo', ''):
	query = {
	'video_id': video_id,
	'ps': 'default',
	'eurl': '',
	'gl': 'US',
	'hl': 'en',
	}
	if el:
	query['el'] = el
	if sts:
	query['sts'] = sts
	video_info_webpage = self._download_webpage(
	'%s://www.youtube.com/get_video_info' % proto,
	video_id, note=False,
	errnote='unable to download video info webpage',
	fatal=False, query=query)

[Happyfeet01]

i have the same problem with my Server. it seem´s ig get blocked :-/

`sshadmin@larsmueller:$ youtube-dl --extract-audio --audio-format mp3 --yes-playlist https://www.youtube.com/watch?v=6VKduh2PCpA&list=RDEMZsvAqlLjps6ZCWsouVUdig
[1] 24901
sshadmin@larsmueller:$ [youtube] 6VKduh2PCpA: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[1]+ Exit 1 youtube-dl --extract-audio --audio-format mp3 --yes-playlist https://www.youtube.com/watch?v=6VKduh2PCpA`

[Turacbey]

Using proxy works for me. I downloads hundreds videos in a hour.
Try to use socks5 or level 1 https proxy.

[bathrobehero]

^Downloading hundreds of videos is not the issue, checking dozens of channels for new videos (even using --download-archive) is where I run into this.

[Turacbey]

^Downloading hundreds of videos is not the issue, checking dozens of channels for new videos (even using --download-archive) is where I run into this.

But all of the requests communicate with YouTube servers. isn't it ?

[bathrobehero]

^No, it downloades all the pages and it skips looking for videos that have been locally downloaded based on the --download-archive lists. It skips thousands of videos a second so no way it does that remotely.

[Happyfeet01]

Using proxy works for me. I downloads hundreds videos in a hour.
Try to use socks5 or level 1 https proxy.

Which Proxy do you use? Any Guide available?

Edit: it works again with an Proxy

[Turacbey]

Using proxy works for me. I downloads hundreds videos in a hour.
Try to use socks5 or level 1 https proxy.

Which Proxy do you use? Any Guide available?

Edit: it works again with an Proxy

Hello,
I use my own proxy created from vds/vps servers. You can create your own socks5 proxy server using ubuntu + Dante.

But there is some threshold for each ip. If you use same ip or proxy a little bit too much, they are banned by YouTube for some hours. So you can need more than 1 ip/proxy if you communicate YouTube servers too much. It is my experience.

[KaMyKaSii]

Happened to me now. I downloaded just one video today, and in the second, from the same channel, I got this error

[debug] System config: []
[debug] User config: ['-o', '/storage/emulated/0/youtube-dl/%(extractor)s/%(title)s-%(id
)s.%(ext)s', '-f', 'webm[height=1080][fps=60]+bestaudio/webm[height=720][fps=60]+bestaud
io/webm[height=1080]+bestaudio/webm[height=720]+bestaudio/mp4[height=1080][fps=60]+besta
udio/mp4[height=720][fps=60]+bestaudio/mp4[height=1080]+bestaudio/mp4[height=720]+bestau
dio/best', '--no-check-certificate', '--no-mtime', '-R', 'infinite']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://youtu.be/Ilg8eIHZR5Q']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.10.29
[debug] Python version 3.7.5 (CPython) - Linux-4.4.141-perf+-aarch64-with-libc
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1
[debug] Proxy map: {}
[youtube] Ilg8eIHZR5Q: Downloading webpage
[youtube] Ilg8eIHZR5Q: Downloading video info webpage
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
ERROR: Ilg8eIHZR5Q: YouTube said: Unable to extract video data
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/youtube_dl/YoutubeDL
.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/youtube_dl/extractor
/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/youtube_dl/extractor
/youtube.py", line 1836, in _real_extract
    'YouTube said: %s' % unavailable_message, expected=True, video_id=video_id)
youtube_dl.utils.ExtractorError: Ilg8eIHZR5Q: YouTube said: Unable to extract video data```

[png-enterprises]

Never had this issue until recently. Was trying to download a big list of videos sequentially and started getting this error after about 15 successful downloads or so.

Seems to dislike the repeated requests to /get_video_info. After I solved a captcha, I was able to view videos in the browser again, but /get_video_info is still blocked.

I get this page, which I've never seen before.

(venv) $ youtube-dl --verbose "https://www.youtube.com/watch?v=BqnG_Ei35JE"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=BqnG_Ei35JE']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.10.29
[debug] Python version 3.6.8 (CPython) - FreeBSD-12.0-RELEASE-p4-amd64-64bit-ELF
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4
[debug] Proxy map: {}
[youtube] BqnG_Ei35JE: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/mnt/Data/Public/Miscellaneous/cloud-locker/home/png/services/youtube-downloader/venv/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mnt/Data/Public/Miscellaneous/cloud-locker/home/png/services/youtube-downloader/venv/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2237, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 564, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

[Happyfeet01]

Never had this issue until recently. Was trying to download a big list of videos sequentially and started getting this error after about 15 successful downloads or so.

Seems to dislike the repeated requests to /get_video_info. After I solved a captcha, I was able to view videos in the browser again, but /get_video_info is still blocked.

I get this page, which I've never seen before.

(venv) $ youtube-dl --verbose "https://www.youtube.com/watch?v=BqnG_Ei35JE"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.youtube.com/watch?v=BqnG_Ei35JE']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.10.29
[debug] Python version 3.6.8 (CPython) - FreeBSD-12.0-RELEASE-p4-amd64-64bit-ELF
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4
[debug] Proxy map: {}
[youtube] BqnG_Ei35JE: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/mnt/Data/Public/Miscellaneous/cloud-locker/home/png/services/youtube-downloader/venv/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/mnt/Data/Public/Miscellaneous/cloud-locker/home/png/services/youtube-downloader/venv/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2237, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 564, in error
    result = self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 756, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Use an Proxy Server and it works fine https://www.netzwelt.de/proxy/index.html

[ZYinMD]

When I get banned, will I get automatically unbanned after sometime?

[Happyfeet01]

When I get banned, will I get automatically unbanned after sometime?

Yes

[ndtreviv]

Using proxy works for me. I downloads hundreds videos in a hour.
Try to use socks5 or level 1 https proxy.

Which Proxy do you use? Any Guide available?
Edit: it works again with an Proxy

Hello,
I use my own proxy created from vds/vps servers. You can create your own socks5 proxy server using ubuntu + Dante.

But there is some threshold for each ip. If you use same ip or proxy a little bit too much, they are banned by YouTube for some hours. So you can need more than 1 ip/proxy if you communicate YouTube servers too much. It is my experience.

I'm curious as to how this works. I have two machines in my office. One gets 429s right now, the other doesn't. Surely as far as youtube are concerned they're coming from the same source. How are they detecting one and not the other?

[Kjarrval]

For some reason, my 429 messages seem to be video specific. I could be attempting to download a video and get a 429 error, then successfully download another video after that, and then get a 429 error when I try the first one again or another one.

[245123]

Using --cookies solves the problem

[ndtreviv]

Using --cookies solves the problem

Please elaborate: using a different value for --cookies for every request, or the same value?

I'm still not sure why it works just fine on my Macbook, but on the Linux box next to me in I just get 429s...
In both cases I'm not using --cookies

[wayneworkman]

I too have this problem. Only work-around I've found is to get a new public IP for traffic to come from. Shortly after that though, it happens again.

[wayneworkman]

With a new public IP, this has fixed it for me.
--sleep-interval 60
I think it'll be fine though, because once I get the channels downloaded, all I will be getting are a couple new videos each day.

[bathrobehero]

^I only use it to download new videos for channels I have hundreds of videos already. It doesn't matter, I still get 429 as youtube-dl is scraping the pages for videos. And --sleep-interval does no effect on that initial scraping, which I think is the issue.

[wayneworkman]

@bathrobehero that's not the issue. They are throttling based on requests to /get_video_info. Getting the video listings always works for me.

[asheshoo]

I have a weird one. Its working for me on Mac OS but fails on linux. Both OSs running v2019.11.28 and operate on the same local network but sharing same public / WAN address.

[RedPenguin2]

I have a strange version of this issue myself.

I had an old PC running Vista that I occasionally captured a Live Stream at times.

Sometimes a regular video at times also.

Yet since Thur, youtube-dl and a similar program both spit out "429 Too Many Requests" even if I give it hours-days.

Yet my Ubuntu PC also with youtube-dl and similar tools, same public IP, never once got the 429 error.

I even tried changing user-agents and everything on Vista, nothing helps.

How could YouTube so easily detect when I am sending from the Vista PC or when I am sending from Ubuntu? It obviously arrives from the same public IP......

[0xattr]

I have a Home-Server and a Workstation both are visible from the Internet with the same public IP. On the Server I get 429s and on my Workstation it just works fine. Tested at the same time with same video. Its kinda funny xD There is deffo more behind it than just "use a proxy!!!" Google is probably using some kind of fingerprinting the client that connects - in the end youtube-dl is just a "web-browser" - and that way they can differentiate between machines. It might be possible to circumvent, but would probbably need some investment of dev time, wich is unlikely to happen. IMO the times of automatic downloads from youtube are over. #neverforget

[Zuccace]

I've done some patching based on this post and it seems to be working most of the time.

By using tor the results are somewhat random as I guess youtube/google blocks tor exit nodes they don't like. But without proxy it is working atm...

[dstftw]

For all complainers over here: there will be no solution until you provide complete reproducible environment. We are not black magicians and can't fix something that does not reproduce on our side. Any useless posts like "any solution?" will be considered spam with corresponding consequences.