[Instagram] HTTPError 429 (Too Many Requests)

[OrthoCube]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.09.12.1
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-c', '--sleep-interval', '5', '-v', 'https://www.instagram.com/tommyrts/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.09.12.1
[debug] Python version 3.7.4 (CPython) - Linux-5.2.13-arch1-1-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1, rtmpdump 2.4
[debug] Proxy map: {}
[instagram:user] tommyrts: Downloading webpage
[download] Downloading playlist: tommyrts
[instagram:user] tommyrts: Downloading JSON page 1
[instagram:user] tommyrts: Downloading JSON page 2
[instagram:user] tommyrts: Downloading JSON page 3
[instagram:user] tommyrts: Downloading JSON page 4
[instagram:user] tommyrts: Downloading JSON page 5
[instagram:user] tommyrts: Downloading JSON page 6
ERROR: Unable to download JSON metadata: HTTP Error 429: - (caused by <HTTPError 429: '-'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 627, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2229, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

Trying to download an Instagram user's videos, youtube-dl tries to get the JSON pages one at a time, causing Instagram to give an HTTP too many requests error eventually. I have seen that youtube-dl fixed this issue by downloading all page data in one request, but that doesn't seem to be the case here.

I am terribly sorry if this is a duplicate issue. If there is something I can do to fix the issue, I would gladly help.

[devmane144]

I ran into this issue as well today in the newest version with youtube (2019.10.16)

[Zuccace]

I wonder if this is related to my issue with the only difference being that I get the error from youtube downloads.

EDIT see this: #21729 (comment)

[OrthoCube]

@devmane144 , @Zuccace , good day.

I was experimenting last night, and I was able to (unofficially and hackish-ly) fix the problem by editing the code.

Add the following code on /usr/lib/python3.7/site-packages/youtube_dl/extractor/common.py (in case of Arch Linux, I don't know about Windows) on line #627 to make it look like this;

        try:
            time.sleep(4)
            return self._downloader.urlopen(url_or_request)

(make sure to follow the indentation, since this is Python)

This is a dirty hack, and adds a 4-second delay before every download, be it metadata download, page download, or actual video download. I tried smaller delays, but 4 works the best. This prevents Instagram from limiting our access. This will also limit downloads from other websites such as YouTube, so this is only a temporary fix and should be removed when not needed. Rather, the developers should turn this into a command switch to enable or disable at will.

Also, another reminder: be sure to add -i switch because sometimes, there will be some 404 Not Found videos on Instagram. Starting from scratch again might take a very long time, so better just ignore videos that can't be downloaded.

Good luck!

Edit: I will try to make a pull request to add a command line switch to add delays before downloading and fix the issue officially and publicly.

[devmane144]

Thanks for getting back! I will give it a go!

[Zuccace]

I knew some delays between requests would solve this.
There should be "--delay" in youtube-dl maybe...

[OrthoCube]

I'm too busy to actually edit the code and submit a PR (plus, I'm not familiar with YTDL codebase) so I think I can't fix this soon. In the moment, I'll reopen the issue, and hopefully someone with more knowledge than me can help fix this officially.