"Could not send HEAD request to https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html: HTTP Error 500: INTERNAL SERVER ERROR"

[benklaasen]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like this: [x])
• Use the Preview tab to see what your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.03.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.03.03

$ youtube-dl --version
2018.03.03

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

• Single video: https://www.youtube.com/watch?v=BaW_jenozKc
• Single video: https://youtu.be/BaW_jenozKc
• Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

---

Hi -

Could you add support for this site, which hosts its videos on Vimeo's streaming platform? The site requires a subscription.

Sample URLs are as follows:

• https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html
• https://learn.miguelgrinberg.com/watch/mega-tutorial/ch02.html
  ...and so on, up to:
• https://learn.miguelgrinberg.com/watch/mega-tutorial/ch23.html

Authenticated users can play videos at those URLs in a browser. Attempting to fetch videos from any of those URLs using youtube-dl fails with the error ERROR: Unable to download webpage: HTTP Error 500: INTERNAL SERVER ERROR (caused by HTTPError()); despite providing credentials on the command line.

The full output from a verbose youtube-dl invocation follows:

$ youtube-dl --verbose -u PRIVATE https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'-u', u'PRIVATE', u'https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.03.03
[debug] Python version 2.7.14 (CPython) - Linux-4.13.0-36-generic-x86_64-with-Ubuntu-17.10-artful
[debug] exe versions: ffmpeg 3.3.4-2, ffprobe 3.3.4-2, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ch01: Requesting header
WARNING: Could not send HEAD request to https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html: HTTP Error 500: INTERNAL SERVER ERROR
[generic] ch01: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 500: INTERNAL SERVER ERROR (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 519, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 2199, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Thank you for taking the time to look at this!

[benklaasen]

I also tried fetching one of the videos using the cookies stored in my browser; this might be a more promising approach.

Thanks again for your time.

 $  youtube-dl --cookies cookies.txt --verbose -u PRIVATE https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--cookies', u'cookies.txt', u'--verbose', u'-u', u'PRIVATE', u'https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html']
Type account password and press [Return]: 
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.03.03
[debug] Python version 2.7.14 (CPython) - Linux-4.13.0-36-generic-x86_64-with-Ubuntu-17.10-artful
[debug] exe versions: ffmpeg 3.3.4-2, ffprobe 3.3.4-2, rtmpdump 2.4
[debug] Proxy map: {}
[generic] ch01: Requesting header
WARNING: Falling back on generic information extractor.
[generic] ch01: Downloading webpage
[generic] ch01: Extracting information
ERROR: Unsupported URL: https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/generic.py", line 2221, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/compat.py", line 2542, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/compat.py", line 2531, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1659, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
    raise err
ParseError: mismatched tag: line 35, column 4
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 785, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/common.py", line 440, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/extractor/generic.py", line 3145, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://learn.miguelgrinberg.com/watch/mega-tutorial/ch01.html

[benklaasen]

@yan12125 What can I do to help you satisfy the prerequisites to be able to address this feature request?