HTTP error 500 on YouTube when trying to download very large playlist

[manoelhortaribeiro]

Checklist

• I'm reporting a broken site support issue
• I've verified that I'm running youtube-dl version 2019.04.30
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

ERROR: Unable to download JSON metadata: Remote end closed connection without response (caused by RemoteDisconnected('Remote end closed connection without response')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 626, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2227, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/site-packages/youtube_dl/utils.py", line 1140, in https_open
    req, **kwargs)
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/urllib/request.py", line 1320, in do_open
    r = h.getresponse()
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/home/manoelribeiro/anaconda3/envs/community_fingerprinting/lib/python3.7/http/client.py", line 265, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"

'NoneType' object is not subscriptable
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.30
[debug] Python version 3.7.1 (CPython) - Linux-4.15.0-47-generic-x86_64-with-debian-buster-sid
[debug] exe versions: ffmpeg 3.4.4, ffprobe 3.4.4, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}

Description

Hey, I'm trying to download just the meta-data of around some playlists with 100,000~ videos. My error seems to be easily reproduced witha command like:

youtube-dl --flat-playlist --skip-download "https://www.youtube.com/watch?v=v0Zwe9Q40iw&list=UUupvZG-5ko_eiXAupbDfxWw"

It goes on to download the pages of the playlist, but, as there are many pages, a (seemingly random) HTTP 500 error eventually appears.

I've tried to get around this by downloading the playlist in chunks, but weirdly, even if you put flags to change where the playlist starts or ends, such as playlist-start or playlist-end, it still has to download the entire playlist.

Any ideas on how to solve this?

Best,
Manoel

[dstftw]

You can't solve server side youtube error. What you can do is to try downloading more granular playlists.

[manoelhortaribeiro]

I think that is unfair... There should be a way of retrying once this server error occurs?

And it also seems bad that you choose to download starting from K and not always from the beggining?